{"expand":"renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id":"38006","self":"https://jira.geedge.net/rest/api/2/issue/38006","key":"OMPUB-1026","fields":{"issuetype":{"self":"https://jira.geedge.net/rest/api/2/issuetype/10004","id":"10004","description":"","iconUrl":"https://jira.geedge.net/secure/viewavatar?size=xsmall&avatarId=10303&avatarType=issuetype","name":"故障","subtask":false,"avatarId":10303},"components":[],"timespent":null,"timeoriginalestimate":null,"description":"1.巡检时发现Hbase，Hadoop服务宕机告警,如附件1-1告警信息所示；\r\n2.反馈至本地大数据同事，经排查Hbase，Hadoop服务正常，监控端口发生变化；\r\n3.问题现状：\r\n（1）Hbase监控图表显示该服务频繁重启，24h内重启次数>10次，如附件1-2,1-3Hbase status所示; 且Hbase监控端口频繁发生变化，这种现状是否存在隐患情况以及如何处理？\r\n  (2)Hadoop监控端口发生变化后，哪吒监控端口还是原来端口，故有一台Hadoop一直显示宕机，是否需要将哪吒上监控端口换为Hadoop现在使用的端口或是使用其他解决方案？","project":{"self":"https://jira.geedge.net/rest/api/2/project/10206","id":"10206","key":"OMPUB","name":"Operation and Maintenance","projectTypeKey":"business","avatarUrls":{"48x48":"https://jira.geedge.net/secure/projectavatar?pid=10206&avatarId=10715","24x24":"https://jira.geedge.net/secure/projectavatar?size=small&pid=10206&avatarId=10715","16x16":"https://jira.geedge.net/secure/projectavatar?size=xsmall&pid=10206&avatarId=10715","32x32":"https://jira.geedge.net/secure/projectavatar?size=medium&pid=10206&avatarId=10715"},"projectCategory":{"self":"https://jira.geedge.net/rest/api/2/projectCategory/10002","id":"10002","description":"系统运维","name":"MaintenanceDev"}},"fixVersions":[],"aggregatetimespent":null,"resolution":{"self":"https://jira.geedge.net/rest/api/2/resolution/10000","id":"10000","description":"该问题的工作流程已完成。","name":"完成"},"timetracking":{},"customfield_10401":null,"customfield_10104":null,"customfield_10402":null,"customfield_10105":"0|i04l9w:","customfield_10403":null,"customfield_10404":null,"attachment":[{"self":"https://jira.geedge.net/rest/api/2/attachment/45414","id":"45414","filename":"1-1告警信息.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-10-08T11:26:46.245+0800","size":117326,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45414/1-1%E5%91%8A%E8%AD%A6%E4%BF%A1%E6%81%AF.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45414/_thumb_45414.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45413","id":"45413","filename":"1-2 Hbase status.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-10-08T11:39:32.315+0800","size":123150,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45413/1-2+Hbase+status.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45413/_thumb_45413.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45412","id":"45412","filename":"1-3 Hbase status.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-10-08T11:39:32.072+0800","size":94506,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45412/1-3+Hbase+status.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45412/_thumb_45412.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45419","id":"45419","filename":"HBase界面.jpg","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-09T12:59:50.868+0800","size":141576,"mimeType":"image/jpeg","content":"https://jira.geedge.net/secure/attachment/45419/HBase%E7%95%8C%E9%9D%A2.jpg","thumbnail":"https://jira.geedge.net/secure/thumbnail/45419/_thumb_45419.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45420","id":"45420","filename":"HDFS界面.jpg","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-09T12:59:50.876+0800","size":137790,"mimeType":"image/jpeg","content":"https://jira.geedge.net/secure/attachment/45420/HDFS%E7%95%8C%E9%9D%A2.jpg","thumbnail":"https://jira.geedge.net/secure/thumbnail/45420/_thumb_45420.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45625","id":"45625","filename":"僵尸进程.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-11T11:21:00.214+0800","size":1117879,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45625/%E5%83%B5%E5%B0%B8%E8%BF%9B%E7%A8%8B.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45625/_thumb_45625.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45475","id":"45475","filename":"进程启动时间.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-09T17:54:30.373+0800","size":996604,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45475/%E8%BF%9B%E7%A8%8B%E5%90%AF%E5%8A%A8%E6%97%B6%E9%97%B4.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45475/_thumb_45475.png"}],"aggregatetimeestimate":null,"resolutiondate":"2023-10-12T13:51:22.325+0800","workratio":-1,"summary":"【XJ-NPM】IDC环境 Hbase、Hadoop服务监控端口异常","lastViewed":null,"watches":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-1026/watchers","watchCount":2,"isWatching":false},"creator":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"subtasks":[],"created":"2023-10-08T19:08:24.153+0800","reporter":{"self":"https://jira.geedge.net/rest/api/2/user?username=sunjiajia","name":"sunjiajia","key":"JIRAUSER10905","emailAddress":"sunjiajia@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10905&avatarId=11804","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10905&avatarId=11804","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10905&avatarId=11804","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10905&avatarId=11804"},"displayName":"孙佳佳","active":true,"timeZone":"Asia/Shanghai"},"customfield_10000":"{summaryBean=com.atlassian.jira.plugin.devstatus.rest.SummaryBean@3e695ea5[summary={pullrequest=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@36c6e389[overall=PullRequestOverallBean{stateCount=0, state='OPEN', details=PullRequestOverallDetails{openCount=0, mergedCount=0, declinedCount=0}},byInstanceType={}], build=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@2d166c4b[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BuildOverallBean@2c4d08cb[failedBuildCount=0,successfulBuildCount=0,unknownBuildCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], review=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@58802b93[overall=com.atlassian.jira.plugin.devstatus.summary.beans.ReviewsOverallBean@75be8681[stateCount=0,state=<null>,dueDate=<null>,overDue=false,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], deployment-environment=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@6bd4d849[overall=com.atlassian.jira.plugin.devstatus.summary.beans.DeploymentOverallBean@2f9b9ea8[topEnvironments=[],showProjects=false,successfulCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], repository=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@769ee47b[overall=com.atlassian.jira.plugin.devstatus.summary.beans.CommitOverallBean@7715eea7[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], branch=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@3a4e669c[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BranchOverallBean@4953d1a3[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}]},errors=[],configErrors=[]], devSummaryJson={\"cachedValue\":{\"errors\":[],\"configErrors\":[],\"summary\":{\"pullrequest\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":\"OPEN\",\"details\":{\"openCount\":0,\"mergedCount\":0,\"declinedCount\":0,\"total\":0},\"open\":true},\"byInstanceType\":{}},\"build\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"failedBuildCount\":0,\"successfulBuildCount\":0,\"unknownBuildCount\":0},\"byInstanceType\":{}},\"review\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":null,\"dueDate\":null,\"overDue\":false,\"completed\":false},\"byInstanceType\":{}},\"deployment-environment\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"topEnvironments\":[],\"showProjects\":false,\"successfulCount\":0},\"byInstanceType\":{}},\"repository\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}},\"branch\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}}}},\"isStale\":false}}","aggregateprogress":{"progress":0,"total":0},"customfield_10100":null,"priority":{"self":"https://jira.geedge.net/rest/api/2/priority/3","iconUrl":"https://jira.geedge.net/images/icons/priorities/medium.svg","name":"Medium","id":"3"},"customfield_10200":null,"customfield_10400":null,"labels":["XJ-CUCC-IDC"],"environment":null,"timeestimate":null,"aggregatetimeoriginalestimate":null,"versions":[],"duedate":"2023-10-12","progress":{"progress":0,"total":0},"issuelinks":[],"comment":{"comments":[{"self":"https://jira.geedge.net/rest/api/2/issue/38006/comment/67583","id":"67583","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"当前状态：\r\n # 该问题与监控端口无关，为231.103服务器HregionServer和DataNode进程异常。\r\n # 目前103数据节点已被移出集群，不影响正常使用。\r\n\r\n \r\n\r\n详细情况：\r\n\r\n1：HregionServer和DataNode进程启动时间在2023年3月15/16号，且没有重启记录。\r\n\r\n!进程启动时间.png|thumbnail!\r\n2：HregionServer最新日志时间是2023年9月22号，同时进入hbase的log目录后使用ll命令会卡死。\r\n\r\n3：231.103上运行的Hadoop和HBase的主节点，均可正常提供服务器，界面可正常访问。\r\n\r\n \r\n\r\n综上：怀疑HregionServer和DataNode进程存在假死或僵尸进程情况，需再确认。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-09T18:01:07.276+0800","updated":"2023-10-09T18:01:07.276+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/38006/comment/67710","id":"67710","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"经后续确认，该问题为HregionServer和DataNode僵尸进程导致，与GAL-248出现僵尸进程时现象一致。\r\n\r\n!僵尸进程.png|thumbnail!\r\n\r\n \r\n\r\n处理方式：通过升级Linux系统内核3.10.0-693.e17.x86_64到kernel-3.10.0-1160.el7.x86_64，重启后进程恢复正常，后续持续观察。\r\n\r\n ","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-11T11:39:26.548+0800","updated":"2023-10-11T11:39:26.548+0800"}],"maxResults":2,"total":2,"startAt":0},"votes":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-1026/votes","votes":0,"hasVoted":false},"worklog":{"startAt":0,"maxResults":20,"total":0,"worklogs":[]},"assignee":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"updated":"2023-10-12T13:51:22.326+0800","status":{"self":"https://jira.geedge.net/rest/api/2/status/5","description":"一项决议已采取了各种措施, 它正在等待验证的记者。这里的问题是重新打开或是关闭的。","iconUrl":"https://jira.geedge.net/images/icons/statuses/resolved.png","name":"已解决","id":"5","statusCategory":{"self":"https://jira.geedge.net/rest/api/2/statuscategory/3","id":3,"key":"done","colorName":"green","name":"完成"}}}}