{"expand":"renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id":"31660","self":"https://jira.geedge.net/rest/api/2/issue/31660","key":"OMPUB-754","fields":{"issuetype":{"self":"https://jira.geedge.net/rest/api/2/issuetype/10004","id":"10004","description":"","iconUrl":"https://jira.geedge.net/secure/viewavatar?size=xsmall&avatarId=10303&avatarType=issuetype","name":"故障","subtask":false,"avatarId":10303},"components":[],"timespent":null,"timeoriginalestimate":null,"description":"新疆电信环境Nezha问题描述\r\n1.开启电信NeZha服务时，72.1会突然卡住；\r\n2.Prometheus服务会自行重启，并且开启服务时，Prometheu进程的cpu占比会特别高；\r\ncortex和loki报错日志情况如图所示。","project":{"self":"https://jira.geedge.net/rest/api/2/project/10206","id":"10206","key":"OMPUB","name":"Operation and Maintenance","projectTypeKey":"business","avatarUrls":{"48x48":"https://jira.geedge.net/secure/projectavatar?pid=10206&avatarId=10715","24x24":"https://jira.geedge.net/secure/projectavatar?size=small&pid=10206&avatarId=10715","16x16":"https://jira.geedge.net/secure/projectavatar?size=xsmall&pid=10206&avatarId=10715","32x32":"https://jira.geedge.net/secure/projectavatar?size=medium&pid=10206&avatarId=10715"},"projectCategory":{"self":"https://jira.geedge.net/rest/api/2/projectCategory/10002","id":"10002","description":"系统运维","name":"MaintenanceDev"}},"fixVersions":[],"aggregatetimespent":null,"resolution":{"self":"https://jira.geedge.net/rest/api/2/resolution/10000","id":"10000","description":"该问题的工作流程已完成。","name":"完成"},"timetracking":{},"customfield_10401":null,"customfield_10104":null,"customfield_10402":null,"customfield_10105":"0|i03irg:","customfield_10403":null,"customfield_10404":null,"attachment":[{"self":"https://jira.geedge.net/rest/api/2/attachment/33897","id":"33897","filename":"cortex报错日志-1.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"created":"2022-12-23T18:25:38.702+0800","size":2017722,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/33897/cortex%E6%8A%A5%E9%94%99%E6%97%A5%E5%BF%97-1.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/33897/_thumb_33897.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/33896","id":"33896","filename":"loki报错日志-1.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"created":"2022-12-23T18:25:36.132+0800","size":1904485,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/33896/loki%E6%8A%A5%E9%94%99%E6%97%A5%E5%BF%97-1.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/33896/_thumb_33896.png"}],"aggregatetimeestimate":null,"resolutiondate":"2022-12-26T14:11:28.993+0800","workratio":-1,"summary":"新疆电信哪吒服务异常和Prometheus服务自行重启","lastViewed":null,"watches":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-754/watchers","watchCount":4,"isWatching":false},"creator":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"subtasks":[],"created":"2022-12-23T18:29:15.750+0800","reporter":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"customfield_10000":"{summaryBean=com.atlassian.jira.plugin.devstatus.rest.SummaryBean@2c6dc4a5[summary={pullrequest=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@2ac8e470[overall=PullRequestOverallBean{stateCount=0, state='OPEN', details=PullRequestOverallDetails{openCount=0, mergedCount=0, declinedCount=0}},byInstanceType={}], build=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@52d32c03[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BuildOverallBean@3f55b94d[failedBuildCount=0,successfulBuildCount=0,unknownBuildCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], review=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@78a3483d[overall=com.atlassian.jira.plugin.devstatus.summary.beans.ReviewsOverallBean@24f45e6a[stateCount=0,state=<null>,dueDate=<null>,overDue=false,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], deployment-environment=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@6f0cddc3[overall=com.atlassian.jira.plugin.devstatus.summary.beans.DeploymentOverallBean@7d819ebe[topEnvironments=[],showProjects=false,successfulCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], repository=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@1f01d393[overall=com.atlassian.jira.plugin.devstatus.summary.beans.CommitOverallBean@1980d917[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], branch=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@106bed21[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BranchOverallBean@616ee5fc[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}]},errors=[],configErrors=[]], devSummaryJson={\"cachedValue\":{\"errors\":[],\"configErrors\":[],\"summary\":{\"pullrequest\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":\"OPEN\",\"details\":{\"openCount\":0,\"mergedCount\":0,\"declinedCount\":0,\"total\":0},\"open\":true},\"byInstanceType\":{}},\"build\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"failedBuildCount\":0,\"successfulBuildCount\":0,\"unknownBuildCount\":0},\"byInstanceType\":{}},\"review\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":null,\"dueDate\":null,\"overDue\":false,\"completed\":false},\"byInstanceType\":{}},\"deployment-environment\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"topEnvironments\":[],\"showProjects\":false,\"successfulCount\":0},\"byInstanceType\":{}},\"repository\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}},\"branch\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}}}},\"isStale\":false}}","aggregateprogress":{"progress":0,"total":0},"customfield_10100":null,"priority":{"self":"https://jira.geedge.net/rest/api/2/priority/1","iconUrl":"https://jira.geedge.net/images/icons/priorities/highest.svg","name":"Highest","id":"1"},"customfield_10200":null,"customfield_10400":null,"labels":["XJ"],"environment":null,"timeestimate":null,"aggregatetimeoriginalestimate":null,"versions":[],"duedate":"2022-12-23","progress":{"progress":0,"total":0},"issuelinks":[],"comment":{"comments":[{"self":"https://jira.geedge.net/rest/api/2/issue/31660/comment/51739","id":"51739","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=shizhendong","name":"shizhendong","key":"JIRAUSER10205","emailAddress":"shizhendong@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?avatarId=10335","24x24":"https://jira.geedge.net/secure/useravatar?size=small&avatarId=10335","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&avatarId=10335","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&avatarId=10335"},"displayName":"史振东","active":true,"timeZone":"Asia/Shanghai"},"body":"BUG 产生的原因：Prometheus wal 目录数据积压严重，造成在启动 prometheus 时进行的 Replay WAL 占用大量内存，是此问题的直接原因。\r\n\r\n产生的现象：由于 prometheus  Replay WAL 过程中占用了大量内存，导致 72.1 服务内存占满，此问题体现在服务器卡顿，大量服务异常。\r\n\r\n如何定位：排查前，现场同事已将 nezha 相关服务 stop，于是在启动 prometheus 服务时，出现 Replaying WAL (349/452) 信息，观察内存占用情况，确认为 Prometheus 组件问题。\r\n\r\n解决方式：将 wal 目录删除并重启 prometheus 服务解决问题。\r\n\r\n存在的其它问题：目前 wal 目录文件数据积压问题并未确认，持续观察中。\r\n\r\n此问题于：2022/12/23 日修复，恢复 NEZHA 系统可用。2022/12/26 日观察 NEZHA 系统 & Prometheus 服务运行均正常，内存使用情况未发现问题。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=shizhendong","name":"shizhendong","key":"JIRAUSER10205","emailAddress":"shizhendong@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?avatarId=10335","24x24":"https://jira.geedge.net/secure/useravatar?size=small&avatarId=10335","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&avatarId=10335","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&avatarId=10335"},"displayName":"史振东","active":true,"timeZone":"Asia/Shanghai"},"created":"2022-12-26T14:11:18.375+0800","updated":"2022-12-26T14:11:18.375+0800"}],"maxResults":1,"total":1,"startAt":0},"votes":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-754/votes","votes":0,"hasVoted":false},"worklog":{"startAt":0,"maxResults":20,"total":0,"worklogs":[]},"assignee":{"self":"https://jira.geedge.net/rest/api/2/user?username=shizhendong","name":"shizhendong","key":"JIRAUSER10205","emailAddress":"shizhendong@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?avatarId=10335","24x24":"https://jira.geedge.net/secure/useravatar?size=small&avatarId=10335","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&avatarId=10335","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&avatarId=10335"},"displayName":"史振东","active":true,"timeZone":"Asia/Shanghai"},"updated":"2022-12-26T14:11:28.998+0800","status":{"self":"https://jira.geedge.net/rest/api/2/status/5","description":"一项决议已采取了各种措施, 它正在等待验证的记者。这里的问题是重新打开或是关闭的。","iconUrl":"https://jira.geedge.net/images/icons/statuses/resolved.png","name":"已解决","id":"5","statusCategory":{"self":"https://jira.geedge.net/rest/api/2/statuscategory/3","id":3,"key":"done","colorName":"green","name":"完成"}}}}