{"expand":"renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id":"27202","self":"https://jira.geedge.net/rest/api/2/issue/27202","key":"OMPUB-492","fields":{"issuetype":{"self":"https://jira.geedge.net/rest/api/2/issuetype/10004","id":"10004","description":"","iconUrl":"https://jira.geedge.net/secure/viewavatar?size=xsmall&avatarId=10303&avatarType=issuetype","name":"故障","subtask":false,"avatarId":10303},"components":[],"timespent":null,"timeoriginalestimate":null,"description":"DIR-IGW 及BJR-IGW 分别在2022-05-18及2022-05-19出现OLAP kafka down告警消息。\r\n\r\n处理进展：\r\n\r\nDIR-IGW 站点查看kafka界面数据，并提供kafka log日志、内存占用、现场配置文件中容器内存限制大小等给研发定位问题，根据现场数据和日志，研发认为可能是程序内存使用过高，超限制被docker干掉了\r\n\r\n根据研发提供的处理方案更新了修改容器限制从17G->25G,删除并重启了容器，目前告警已消除，解决故障\r\n\r\n \r\n\r\nBJR-IGW 根据现场数据，研发定位故障原因和DIR-IGW一致，故障暂通过重启docker解决。","project":{"self":"https://jira.geedge.net/rest/api/2/project/10206","id":"10206","key":"OMPUB","name":"Operation and Maintenance","projectTypeKey":"business","avatarUrls":{"48x48":"https://jira.geedge.net/secure/projectavatar?pid=10206&avatarId=10715","24x24":"https://jira.geedge.net/secure/projectavatar?size=small&pid=10206&avatarId=10715","16x16":"https://jira.geedge.net/secure/projectavatar?size=xsmall&pid=10206&avatarId=10715","32x32":"https://jira.geedge.net/secure/projectavatar?size=medium&pid=10206&avatarId=10715"},"projectCategory":{"self":"https://jira.geedge.net/rest/api/2/projectCategory/10002","id":"10002","description":"系统运维","name":"MaintenanceDev"}},"fixVersions":[],"aggregatetimespent":null,"resolution":{"self":"https://jira.geedge.net/rest/api/2/resolution/10000","id":"10000","description":"该问题的工作流程已完成。","name":"完成"},"timetracking":{},"customfield_10401":null,"customfield_10104":null,"customfield_10402":null,"customfield_10105":"0|i02tjo:","customfield_10403":null,"customfield_10404":null,"attachment":[{"self":"https://jira.geedge.net/rest/api/2/attachment/28527","id":"28527","filename":"BJR-IGW和DIR-IGW Kafka内存使用.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2022-06-09T11:05:06.424+0800","size":17682,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/28527/BJR-IGW%E5%92%8CDIR-IGW+Kafka%E5%86%85%E5%AD%98%E4%BD%BF%E7%94%A8.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/28527/_thumb_28527.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/28526","id":"28526","filename":"BJR-IGW日志量.jpg","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2022-06-09T11:01:05.911+0800","size":99121,"mimeType":"image/jpeg","content":"https://jira.geedge.net/secure/attachment/28526/BJR-IGW%E6%97%A5%E5%BF%97%E9%87%8F.jpg","thumbnail":"https://jira.geedge.net/secure/thumbnail/28526/_thumb_28526.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/29109","id":"29109","filename":"image-2022-06-27-10-19-08-526.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2022-06-27T10:19:08.451+0800","size":202672,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/29109/image-2022-06-27-10-19-08-526.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/29109/_thumb_29109.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/29107","id":"29107","filename":"kafka-log-timeout.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2022-06-27T09:33:23.998+0800","size":106798,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/29107/kafka-log-timeout.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/29107/_thumb_29107.png"}],"aggregatetimeestimate":null,"resolutiondate":"2022-07-08T09:14:15.130+0800","workratio":-1,"summary":"【E21-olap】DIR-IGW 、BJR-IGW 近两天出现OLAP kafka down 告警","lastViewed":null,"watches":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-492/watchers","watchCount":2,"isWatching":false},"creator":{"self":"https://jira.geedge.net/rest/api/2/user?username=liuju","name":"liuju","key":"JIRAUSER10222","emailAddress":"liuju@zdjizhi.com","avatarUrls":{"48x48":"https://www.gravatar.com/avatar/de39e01c583621fe2030d723f55e0e79?d=mm&s=48","24x24":"https://www.gravatar.com/avatar/de39e01c583621fe2030d723f55e0e79?d=mm&s=24","16x16":"https://www.gravatar.com/avatar/de39e01c583621fe2030d723f55e0e79?d=mm&s=16","32x32":"https://www.gravatar.com/avatar/de39e01c583621fe2030d723f55e0e79?d=mm&s=32"},"displayName":"刘菊","active":false,"timeZone":"Asia/Shanghai"},"subtasks":[],"created":"2022-05-19T16:15:32.902+0800","reporter":{"self":"https://jira.geedge.net/rest/api/2/user?username=liuju","name":"liuju","key":"JIRAUSER10222","emailAddress":"liuju@zdjizhi.com","avatarUrls":{"48x48":"https://www.gravatar.com/avatar/de39e01c583621fe2030d723f55e0e79?d=mm&s=48","24x24":"https://www.gravatar.com/avatar/de39e01c583621fe2030d723f55e0e79?d=mm&s=24","16x16":"https://www.gravatar.com/avatar/de39e01c583621fe2030d723f55e0e79?d=mm&s=16","32x32":"https://www.gravatar.com/avatar/de39e01c583621fe2030d723f55e0e79?d=mm&s=32"},"displayName":"刘菊","active":false,"timeZone":"Asia/Shanghai"},"customfield_10000":"{summaryBean=com.atlassian.jira.plugin.devstatus.rest.SummaryBean@7e72653c[summary={pullrequest=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@23158b8b[overall=PullRequestOverallBean{stateCount=0, state='OPEN', details=PullRequestOverallDetails{openCount=0, mergedCount=0, declinedCount=0}},byInstanceType={}], build=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@56051d80[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BuildOverallBean@59c38d18[failedBuildCount=0,successfulBuildCount=0,unknownBuildCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], review=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@c51f9c2[overall=com.atlassian.jira.plugin.devstatus.summary.beans.ReviewsOverallBean@296555ab[stateCount=0,state=<null>,dueDate=<null>,overDue=false,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], deployment-environment=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@57f2b41a[overall=com.atlassian.jira.plugin.devstatus.summary.beans.DeploymentOverallBean@129c48b1[topEnvironments=[],showProjects=false,successfulCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], repository=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@11003cc9[overall=com.atlassian.jira.plugin.devstatus.summary.beans.CommitOverallBean@124bfb4f[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], branch=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@505bdf78[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BranchOverallBean@2072dc02[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}]},errors=[],configErrors=[]], devSummaryJson={\"cachedValue\":{\"errors\":[],\"configErrors\":[],\"summary\":{\"pullrequest\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":\"OPEN\",\"details\":{\"openCount\":0,\"mergedCount\":0,\"declinedCount\":0,\"total\":0},\"open\":true},\"byInstanceType\":{}},\"build\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"failedBuildCount\":0,\"successfulBuildCount\":0,\"unknownBuildCount\":0},\"byInstanceType\":{}},\"review\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":null,\"dueDate\":null,\"overDue\":false,\"completed\":false},\"byInstanceType\":{}},\"deployment-environment\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"topEnvironments\":[],\"showProjects\":false,\"successfulCount\":0},\"byInstanceType\":{}},\"repository\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}},\"branch\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}}}},\"isStale\":false}}","aggregateprogress":{"progress":0,"total":0},"customfield_10100":null,"priority":{"self":"https://jira.geedge.net/rest/api/2/priority/3","iconUrl":"https://jira.geedge.net/images/icons/priorities/medium.svg","name":"Medium","id":"3"},"customfield_10200":null,"customfield_10400":null,"labels":["E21现场"],"environment":null,"timeestimate":null,"aggregatetimeoriginalestimate":null,"versions":[],"duedate":null,"progress":{"progress":0,"total":0},"issuelinks":[],"comment":{"comments":[{"self":"https://jira.geedge.net/rest/api/2/issue/27202/comment/42376","id":"42376","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"1：根据当时反馈的Kafka、Zookeeper日志没有发现明显的错误信息。\r\n2：通过数据量监控未发现有日志量激增的情况。\r\n !BJR-IGW日志量.jpg|thumbnail! \r\n3：恢复后观察此量点的内存使用，均在4/6GB左右 未再超出上限。\r\n !BJR-IGW和DIR-IGW Kafka内存使用.png|thumbnail! \r\n\r\n因未明确定位是何问题造成的，暂不对所有局点的Kafka容器进行修改，后续持续观察分析。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2022-06-09T11:05:28.257+0800","updated":"2022-06-09T11:06:08.535+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/27202/comment/43426","id":"43426","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"根据现场回传日志查看：\r\n1：Kafka存在与Zookeeper连接超时的情况，与Zookeeper连接超时，kakfa无法及时更新元信息，导致了Kafka服务终止。\r\n !kafka-log-timeout.png|thumbnail! \r\n2：通过查看机器的IO使用率，在Kafka出现连接超时的时间点附近，IO突增且持续；与之前正常情况下的IO使用率曲线有较大差别。\r\n !image-2022-06-27-10-19-08-526.png|thumbnail! \r\n\r\n\r\n解决方案：\r\n1：增加与Zookeeper的超时时间，减少数据刷盘前在内存内缓存的最大时间与大小。\r\n2：在下次更新时，对所有局点的kafka进行配置优化。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2022-06-27T09:39:16.254+0800","updated":"2022-06-27T10:22:34.366+0800"}],"maxResults":2,"total":2,"startAt":0},"votes":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-492/votes","votes":0,"hasVoted":false},"worklog":{"startAt":0,"maxResults":20,"total":0,"worklogs":[]},"assignee":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"updated":"2023-02-02T15:12:29.203+0800","status":{"self":"https://jira.geedge.net/rest/api/2/status/10103","description":"这一问题被认为是完成, 这项决议是正确的。问题已关闭可以重新开放。","iconUrl":"https://jira.geedge.net/images/icons/statuses/generic.png","name":"已关闭","id":"10103","statusCategory":{"self":"https://jira.geedge.net/rest/api/2/statuscategory/3","id":3,"key":"done","colorName":"green","name":"完成"}}}}