{"expand":"renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id":"38167","self":"https://jira.geedge.net/rest/api/2/issue/38167","key":"OMPUB-1031","fields":{"issuetype":{"self":"https://jira.geedge.net/rest/api/2/issuetype/10004","id":"10004","description":"","iconUrl":"https://jira.geedge.net/secure/viewavatar?size=xsmall&avatarId=10303&avatarType=issuetype","name":"故障","subtask":false,"avatarId":10303},"components":[],"timespent":null,"timeoriginalestimate":null,"description":"OLAP在升级23.07版本后，Clickhouse服务器Server43、Server44、Server47、Server48、Server49的/data目录使用率超过80%，其余Clickhouse服务器/data使用率为40%--60%。","project":{"self":"https://jira.geedge.net/rest/api/2/project/10206","id":"10206","key":"OMPUB","name":"Operation and Maintenance","projectTypeKey":"business","avatarUrls":{"48x48":"https://jira.geedge.net/secure/projectavatar?pid=10206&avatarId=10715","24x24":"https://jira.geedge.net/secure/projectavatar?size=small&pid=10206&avatarId=10715","16x16":"https://jira.geedge.net/secure/projectavatar?size=xsmall&pid=10206&avatarId=10715","32x32":"https://jira.geedge.net/secure/projectavatar?size=medium&pid=10206&avatarId=10715"},"projectCategory":{"self":"https://jira.geedge.net/rest/api/2/projectCategory/10002","id":"10002","description":"系统运维","name":"MaintenanceDev"}},"fixVersions":[],"aggregatetimespent":null,"resolution":{"self":"https://jira.geedge.net/rest/api/2/resolution/10000","id":"10000","description":"该问题的工作流程已完成。","name":"完成"},"timetracking":{},"customfield_10401":null,"customfield_10104":null,"customfield_10402":null,"customfield_10105":"0|i04m9o:","customfield_10403":null,"customfield_10404":null,"attachment":[{"self":"https://jira.geedge.net/rest/api/2/attachment/51963","id":"51963","filename":"CK磁盘使用率-20240221.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2024-02-22T11:24:18.511+0800","size":56658,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/51963/CK%E7%A3%81%E7%9B%98%E4%BD%BF%E7%94%A8%E7%8E%87-20240221.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/51963/_thumb_51963.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45812","id":"45812","filename":"query.desc.session_record_local.txt","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-13T09:57:03.598+0800","size":21730,"mimeType":"text/plain","content":"https://jira.geedge.net/secure/attachment/45812/query.desc.session_record_local.txt"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45799","id":"45799","filename":"sdb1.jpg","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=songlongkun","name":"songlongkun","key":"JIRAUSER10914","emailAddress":"songlongkun@geedgenetworks.com","avatarUrls":{"48x48":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=48","24x24":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=24","16x16":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=16","32x32":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=32"},"displayName":"宋龙坤","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-10-12T21:17:16.556+0800","size":47893,"mimeType":"image/jpeg","content":"https://jira.geedge.net/secure/attachment/45799/sdb1.jpg","thumbnail":"https://jira.geedge.net/secure/thumbnail/45799/_thumb_45799.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/51962","id":"51962","filename":"TTL监控.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2024-02-22T11:19:28.171+0800","size":84491,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/51962/TTL%E7%9B%91%E6%8E%A7.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/51962/_thumb_51962.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45809","id":"45809","filename":"存储配额趋势图.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-13T09:53:10.575+0800","size":17121,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45809/%E5%AD%98%E5%82%A8%E9%85%8D%E9%A2%9D%E8%B6%8B%E5%8A%BF%E5%9B%BE.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45809/_thumb_45809.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45811","id":"45811","filename":"日志量（总量）.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-13T09:55:00.567+0800","size":181129,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45811/%E6%97%A5%E5%BF%97%E9%87%8F%EF%BC%88%E6%80%BB%E9%87%8F%EF%BC%89.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45811/_thumb_45811.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45810","id":"45810","filename":"异常节点数据片时间图.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-13T09:53:10.617+0800","size":105251,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45810/%E5%BC%82%E5%B8%B8%E8%8A%82%E7%82%B9%E6%95%B0%E6%8D%AE%E7%89%87%E6%97%B6%E9%97%B4%E5%9B%BE.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45810/_thumb_45810.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/45813","id":"45813","filename":"正常节点数据片时间图.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-13T10:02:50.285+0800","size":36164,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/45813/%E6%AD%A3%E5%B8%B8%E8%8A%82%E7%82%B9%E6%95%B0%E6%8D%AE%E7%89%87%E6%97%B6%E9%97%B4%E5%9B%BE.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/45813/_thumb_45813.png"}],"aggregatetimeestimate":null,"resolutiondate":"2024-02-26T10:41:43.588+0800","workratio":-1,"summary":"【E21现场】OLAP在升级23.07版本后，部分Clickhouse服务器使用率超80%","lastViewed":null,"watches":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-1031/watchers","watchCount":4,"isWatching":false},"creator":{"self":"https://jira.geedge.net/rest/api/2/user?username=songlongkun","name":"songlongkun","key":"JIRAUSER10914","emailAddress":"songlongkun@geedgenetworks.com","avatarUrls":{"48x48":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=48","24x24":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=24","16x16":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=16","32x32":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=32"},"displayName":"宋龙坤","active":true,"timeZone":"Asia/Shanghai"},"subtasks":[],"created":"2023-10-12T21:13:20.301+0800","reporter":{"self":"https://jira.geedge.net/rest/api/2/user?username=songlongkun","name":"songlongkun","key":"JIRAUSER10914","emailAddress":"songlongkun@geedgenetworks.com","avatarUrls":{"48x48":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=48","24x24":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=24","16x16":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=16","32x32":"https://www.gravatar.com/avatar/643f4935f43167fd773a6b701f9fd05b?d=mm&s=32"},"displayName":"宋龙坤","active":true,"timeZone":"Asia/Shanghai"},"customfield_10000":"{summaryBean=com.atlassian.jira.plugin.devstatus.rest.SummaryBean@27d3f48e[summary={pullrequest=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@7c5b5a72[overall=PullRequestOverallBean{stateCount=0, state='OPEN', details=PullRequestOverallDetails{openCount=0, mergedCount=0, declinedCount=0}},byInstanceType={}], build=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@7eeac9e6[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BuildOverallBean@58c49483[failedBuildCount=0,successfulBuildCount=0,unknownBuildCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], review=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@c84a097[overall=com.atlassian.jira.plugin.devstatus.summary.beans.ReviewsOverallBean@63dfe7dd[stateCount=0,state=<null>,dueDate=<null>,overDue=false,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], deployment-environment=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@4bd2b7b9[overall=com.atlassian.jira.plugin.devstatus.summary.beans.DeploymentOverallBean@37ce42cc[topEnvironments=[],showProjects=false,successfulCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], repository=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@1ee7ccba[overall=com.atlassian.jira.plugin.devstatus.summary.beans.CommitOverallBean@29a21b4[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], branch=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@5cab2121[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BranchOverallBean@19b6c05a[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}]},errors=[],configErrors=[]], devSummaryJson={\"cachedValue\":{\"errors\":[],\"configErrors\":[],\"summary\":{\"pullrequest\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":\"OPEN\",\"details\":{\"openCount\":0,\"mergedCount\":0,\"declinedCount\":0,\"total\":0},\"open\":true},\"byInstanceType\":{}},\"build\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"failedBuildCount\":0,\"successfulBuildCount\":0,\"unknownBuildCount\":0},\"byInstanceType\":{}},\"review\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":null,\"dueDate\":null,\"overDue\":false,\"completed\":false},\"byInstanceType\":{}},\"deployment-environment\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"topEnvironments\":[],\"showProjects\":false,\"successfulCount\":0},\"byInstanceType\":{}},\"repository\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}},\"branch\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}}}},\"isStale\":false}}","aggregateprogress":{"progress":0,"total":0},"customfield_10100":null,"priority":{"self":"https://jira.geedge.net/rest/api/2/priority/3","iconUrl":"https://jira.geedge.net/images/icons/priorities/medium.svg","name":"Medium","id":"3"},"customfield_10200":null,"customfield_10400":null,"labels":["Galaxy"],"environment":null,"timeestimate":null,"aggregatetimeoriginalestimate":null,"versions":[],"duedate":null,"progress":{"progress":0,"total":0},"issuelinks":[{"id":"34562","self":"https://jira.geedge.net/rest/api/2/issueLink/34562","type":{"id":"10002","name":"Duplicate","inward":"is duplicated by","outward":"duplicates","self":"https://jira.geedge.net/rest/api/2/issueLinkType/10002"},"inwardIssue":{"id":"38107","key":"OMPUB-1027","self":"https://jira.geedge.net/rest/api/2/issue/38107","fields":{"summary":"【E21现场】完成23.07升级后BOL-IGW站点TSG-OS告警数量增加","status":{"self":"https://jira.geedge.net/rest/api/2/status/10103","description":"这一问题被认为是完成, 这项决议是正确的。问题已关闭可以重新开放。","iconUrl":"https://jira.geedge.net/images/icons/statuses/generic.png","name":"已关闭","id":"10103","statusCategory":{"self":"https://jira.geedge.net/rest/api/2/statuscategory/3","id":3,"key":"done","colorName":"green","name":"完成"}},"priority":{"self":"https://jira.geedge.net/rest/api/2/priority/3","iconUrl":"https://jira.geedge.net/images/icons/priorities/medium.svg","name":"Medium","id":"3"},"issuetype":{"self":"https://jira.geedge.net/rest/api/2/issuetype/10004","id":"10004","description":"","iconUrl":"https://jira.geedge.net/secure/viewavatar?size=xsmall&avatarId=10303&avatarType=issuetype","name":"故障","subtask":false,"avatarId":10303}}}}],"comment":{"comments":[{"self":"https://jira.geedge.net/rest/api/2/issue/38167/comment/67874","id":"67874","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"现场排查：\r\n # 数据情况：\r\n ## Server35/36为Query节点，不存储数据，因此磁盘使用率较低。\r\n ## 对比存储配额趋势图，较升级前（2023-10-06）每天多3TB左右的数据。\r\n ### 其原因为22.11版本分中心ETL汇聚存在丢日志的情况，升级23.07后程序优化，除夜间最高峰时期（总量85w/s）会存在3-5%的日志丢失，其余时间均正常。\r\n ### !存储配额趋势图.png|thumbnail!!日志量（总量）.png|thumbnail!\r\n # 存储配置情况：\r\n ## 现场存储配额：\r\n ### Max Days设置为40天。\r\n ### Security Events表TTL为40天，其余为30天，原因参考：OMPUB-860\r\n ## 对比Clickhouse表数据块时间：\r\n ### Server43/44/47/48/49异常节点数据范围均为40天。\r\n #### !异常节点数据片时间图.png|thumbnail!\r\n ### 对比Server37/38正常节点，session_record表数据范围为30天，security_event表数据范围为40天。\r\n #### !正常节点数据片时间图.png|thumbnail!\r\n ## 通过Clickhouse查询表信息，Server43/44/47/48/49 session_record表均携带30天的TTL配置，参考：[^query.desc.session_record_local.txt]\r\n\r\n \r\n\r\n综上，怀疑为表TTL在部分节点未成功删除数据。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-13T10:07:40.880+0800","updated":"2023-10-13T10:07:40.880+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/38167/comment/67951","id":"67951","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"目前已确认为Server43/44/47/48/49服务器Clickhouse未执行TTL，因TTL功能为组件自控制，未执行原因待后续继续排查，任务暂降级为Medium。\r\n\r\n \r\n\r\n临时处置方式：通过手动触发的方式将Server43/44/47/48/49服务器session_record表，超过TTL配置时间（30天）的数据删除。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-10-13T17:50:41.508+0800","updated":"2023-10-20T09:42:56.015+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/38167/comment/68997","id":"68997","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"经过近期对比排查：\r\n * 选取Server 43节点进行手动触发 *session_record表单分区* 的TTL操作，执行后该节点session_record表恢复TTL功能。\r\n ** 后对Server 44节点进行上述重复操作，未恢复。\r\n * 异常Clickhouse节点{*}也在执行TTL删除操作{*}，但比其他正常节点慢，且{*}无法完全删除{*}全部的过期数据。\r\n ** 例如：有300GB的文件需要删除，只删除了200GB。\r\n * 整体观察Clickhouse集群数据节点，每个节点的数据摄入量、Merge频率没有较大差异。\r\n\r\n后续继续排查，以及确认是否为Clickhouse TTL机制存在相关BUG。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-11-03T11:08:21.022+0800","updated":"2023-11-03T11:12:31.446+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/38167/comment/69937","id":"69937","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"目前现场保留Server 47和Server 49两个节点持续观察，其余节点TTL功能均已恢复正常。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2023-11-24T19:13:29.431+0800","updated":"2023-11-24T19:13:29.431+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/38167/comment/72246","id":"72246","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"后经过测试已复现该问题，为Clickhouse 21.x版本BUG，详细内容见：[Clickhouse TTL失效问题排查|https://docs.geedge.net/pages/viewpage.action?pageId=124752639]","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2024-01-04T10:05:56.047+0800","updated":"2024-01-04T10:05:56.047+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/38167/comment/74859","id":"74859","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"body":"后续操作：在NZ → Componet Status → Clickhouse Status中增加了未删除数据块个数相关监控图表，如图：\r\n\r\n!TTL监控.png|thumbnail!\r\n\r\n \r\n\r\n目前情况：Clickhouse节点使用率基本保持在35%左右，保留的Server 47和Server 49两个节点在50%左右，如下图：\r\n\r\n!CK磁盘使用率-20240221.png|thumbnail!\r\n\r\n \r\n\r\n由于该问题是由Clickhouse 21.x版本BUG引起，需升级版本解决；目前磁盘使用率相对较低，流量稳定的情况下使用率将维持在这个范围；暂将bug状态改为持续追踪状态。","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"created":"2024-02-22T11:28:20.154+0800","updated":"2024-02-22T11:28:20.154+0800"}],"maxResults":6,"total":6,"startAt":0},"votes":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-1031/votes","votes":0,"hasVoted":false},"worklog":{"startAt":0,"maxResults":20,"total":0,"worklogs":[]},"assignee":{"self":"https://jira.geedge.net/rest/api/2/user?username=qidaijie","name":"qidaijie","key":"JIRAUSER10135","emailAddress":"qidaijie@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10135&avatarId=10727","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10135&avatarId=10727","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10135&avatarId=10727","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10135&avatarId=10727"},"displayName":"戚岱杰","active":false,"timeZone":"Asia/Shanghai"},"updated":"2024-02-26T10:41:43.588+0800","status":{"self":"https://jira.geedge.net/rest/api/2/status/5","description":"一项决议已采取了各种措施, 它正在等待验证的记者。这里的问题是重新打开或是关闭的。","iconUrl":"https://jira.geedge.net/images/icons/statuses/resolved.png","name":"已解决","id":"5","statusCategory":{"self":"https://jira.geedge.net/rest/api/2/statuscategory/3","id":3,"key":"done","colorName":"green","name":"完成"}}}}