{"expand":"renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id":"32302","self":"https://jira.geedge.net/rest/api/2/issue/32302","key":"OMPUB-790","fields":{"issuetype":{"self":"https://jira.geedge.net/rest/api/2/issuetype/10004","id":"10004","description":"","iconUrl":"https://jira.geedge.net/secure/viewavatar?size=xsmall&avatarId=10303&avatarType=issuetype","name":"故障","subtask":false,"avatarId":10303},"components":[],"timespent":null,"timeoriginalestimate":null,"description":"{*}X主要丢包服务器IP{*}：\r\n192.168.16.1-3\r\n192.168.17.2-3\r\n192.168.18.1-2\r\n\r\n{*}主要丢包时段{*}：\r\n18:40 —— 22:40\r\n\r\n2023/01/31通过研发协助，定位到丢包问题有一部分原因来自于运营商给系统的流量中包含一部分他们设备内部的通信流量（SYSLOG），2023/02/01已将大部分SYSLOG流量的二元组在汇聚分流上丢弃。\r\n后续增加了重点丢包服务器的sapp绑定核，观察福建Nezha发现仍然丢包，2023/02/03 21:00 对重点服务器进行丢包分析，发现丢包时单个线程的bps和pps均不高，但仍有丢包，通过perf -top -C 100查看也没有占用CPU过高，下图为当时monit_stream -lHs --per-stream截图\r\n!image-2023-02-06-09-43-21-762.png|thumbnail!","project":{"self":"https://jira.geedge.net/rest/api/2/project/10206","id":"10206","key":"OMPUB","name":"Operation and Maintenance","projectTypeKey":"business","avatarUrls":{"48x48":"https://jira.geedge.net/secure/projectavatar?pid=10206&avatarId=10715","24x24":"https://jira.geedge.net/secure/projectavatar?size=small&pid=10206&avatarId=10715","16x16":"https://jira.geedge.net/secure/projectavatar?size=xsmall&pid=10206&avatarId=10715","32x32":"https://jira.geedge.net/secure/projectavatar?size=medium&pid=10206&avatarId=10715"},"projectCategory":{"self":"https://jira.geedge.net/rest/api/2/projectCategory/10002","id":"10002","description":"系统运维","name":"MaintenanceDev"}},"fixVersions":[],"aggregatetimespent":null,"resolution":{"self":"https://jira.geedge.net/rest/api/2/resolution/10001","id":"10001","description":"This issue won't be actioned.","name":"Won't Do"},"timetracking":{},"customfield_10401":null,"customfield_10104":null,"customfield_10402":null,"customfield_10105":"0|i03mtw:","customfield_10403":null,"customfield_10404":null,"attachment":[{"self":"https://jira.geedge.net/rest/api/2/attachment/34755","id":"34755","filename":"image-2023-02-06-09-43-21-762.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=zhangzhihan","name":"zhangzhihan","key":"JIRAUSER10111","emailAddress":"zhangzhihan@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10111&avatarId=12001","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10111&avatarId=12001","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10111&avatarId=12001","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10111&avatarId=12001"},"displayName":"张智涵","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-02-06T09:43:21.935+0800","size":165279,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/34755/image-2023-02-06-09-43-21-762.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/34755/_thumb_34755.png"},{"self":"https://jira.geedge.net/rest/api/2/attachment/34764","id":"34764","filename":"screenshot-1.png","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=zhangzhihan","name":"zhangzhihan","key":"JIRAUSER10111","emailAddress":"zhangzhihan@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10111&avatarId=12001","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10111&avatarId=12001","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10111&avatarId=12001","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10111&avatarId=12001"},"displayName":"张智涵","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-02-06T13:43:47.175+0800","size":379577,"mimeType":"image/png","content":"https://jira.geedge.net/secure/attachment/34764/screenshot-1.png","thumbnail":"https://jira.geedge.net/secure/thumbnail/34764/_thumb_34764.png"}],"aggregatetimeestimate":null,"resolutiondate":"2023-03-14T11:09:30.208+0800","workratio":-1,"summary":"福建项目：福州城域网服务器sapp丢包问题","lastViewed":null,"watches":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-790/watchers","watchCount":2,"isWatching":false},"creator":{"self":"https://jira.geedge.net/rest/api/2/user?username=zhangzhihan","name":"zhangzhihan","key":"JIRAUSER10111","emailAddress":"zhangzhihan@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10111&avatarId=12001","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10111&avatarId=12001","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10111&avatarId=12001","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10111&avatarId=12001"},"displayName":"张智涵","active":true,"timeZone":"Asia/Shanghai"},"subtasks":[],"created":"2023-02-06T09:43:27.425+0800","reporter":{"self":"https://jira.geedge.net/rest/api/2/user?username=zhangzhihan","name":"zhangzhihan","key":"JIRAUSER10111","emailAddress":"zhangzhihan@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10111&avatarId=12001","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10111&avatarId=12001","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10111&avatarId=12001","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10111&avatarId=12001"},"displayName":"张智涵","active":true,"timeZone":"Asia/Shanghai"},"customfield_10000":"{summaryBean=com.atlassian.jira.plugin.devstatus.rest.SummaryBean@7b124392[summary={pullrequest=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@c04f143[overall=PullRequestOverallBean{stateCount=0, state='OPEN', details=PullRequestOverallDetails{openCount=0, mergedCount=0, declinedCount=0}},byInstanceType={}], build=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@603edb8e[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BuildOverallBean@4b9ba023[failedBuildCount=0,successfulBuildCount=0,unknownBuildCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], review=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@28b0bdb[overall=com.atlassian.jira.plugin.devstatus.summary.beans.ReviewsOverallBean@43ac4ad9[stateCount=0,state=<null>,dueDate=<null>,overDue=false,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], deployment-environment=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@55581b89[overall=com.atlassian.jira.plugin.devstatus.summary.beans.DeploymentOverallBean@43d423b9[topEnvironments=[],showProjects=false,successfulCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], repository=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@195b3dcb[overall=com.atlassian.jira.plugin.devstatus.summary.beans.CommitOverallBean@380056bb[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], branch=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@2d918661[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BranchOverallBean@4f3aa181[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}]},errors=[],configErrors=[]], devSummaryJson={\"cachedValue\":{\"errors\":[],\"configErrors\":[],\"summary\":{\"pullrequest\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":\"OPEN\",\"details\":{\"openCount\":0,\"mergedCount\":0,\"declinedCount\":0,\"total\":0},\"open\":true},\"byInstanceType\":{}},\"build\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"failedBuildCount\":0,\"successfulBuildCount\":0,\"unknownBuildCount\":0},\"byInstanceType\":{}},\"review\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":null,\"dueDate\":null,\"overDue\":false,\"completed\":false},\"byInstanceType\":{}},\"deployment-environment\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"topEnvironments\":[],\"showProjects\":false,\"successfulCount\":0},\"byInstanceType\":{}},\"repository\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}},\"branch\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}}}},\"isStale\":false}}","aggregateprogress":{"progress":0,"total":0},"customfield_10100":null,"priority":{"self":"https://jira.geedge.net/rest/api/2/priority/3","iconUrl":"https://jira.geedge.net/images/icons/priorities/medium.svg","name":"Medium","id":"3"},"customfield_10200":null,"customfield_10400":null,"labels":["FUJIAN","TSG-OS性能"],"environment":null,"timeestimate":null,"aggregatetimeoriginalestimate":null,"versions":[],"duedate":null,"progress":{"progress":0,"total":0},"issuelinks":[],"comment":{"comments":[{"self":"https://jira.geedge.net/rest/api/2/issue/32302/comment/53138","id":"53138","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=yangwei","name":"yangwei","key":"JIRAUSER10103","emailAddress":"yangwei@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10103&avatarId=10708","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10103&avatarId=10708","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10103&avatarId=10708","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10103&avatarId=10708"},"displayName":"杨威","active":true,"timeZone":"Asia/Shanghai"},"body":"2023年02月06日上午更新，请[~zhangzhihan] 补充下测试的IP等信息：\r\n * 现象\r\n ** 抽样福州移动两台（16.2-3），使用monit_stream进行观察，发现丢包线程集中在numa node 1的高位CPU（core id 大于110），白天观察，出现频率不是十分稳定\r\n ** 使用sapp和sysinfo.log加top结果定位，初步定位丢包原因为CPU使用接近100%，其中sys us较高（>20%）\r\n ** 使用perf观察对应的core，怀疑原因有两个：\r\n ### hos_client上传时系统调用过高\r\n ### ddos_sketch上报时系统调用过高\r\n * 操作\r\n ** 找了两台机器，分别关闭了hos_client，和ddos_sketch插件，继续观察后续丢包情况","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=yangwei","name":"yangwei","key":"JIRAUSER10103","emailAddress":"yangwei@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10103&avatarId=10708","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10103&avatarId=10708","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10103&avatarId=10708","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10103&avatarId=10708"},"displayName":"杨威","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-02-06T13:26:49.539+0800","updated":"2023-02-06T17:00:12.602+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/32302/comment/53140","id":"53140","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=zhangzhihan","name":"zhangzhihan","key":"JIRAUSER10111","emailAddress":"zhangzhihan@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10111&avatarId=12001","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10111&avatarId=12001","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10111&avatarId=12001","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10111&avatarId=12001"},"displayName":"张智涵","active":true,"timeZone":"Asia/Shanghai"},"body":"192.168.16.3\r\nHos：关闭\r\nddos_sketch：开启\r\n\r\n192.168.16.2\r\nHos：开启\r\nddos_sketch：关闭\r\n\r\n两台sapp丢包目前都有所改善。对比monit_stream，发现192.168.16.3关闭hos后，高位CPUsys使用明显降低\r\n !screenshot-1.png|thumbnail! ","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=zhangzhihan","name":"zhangzhihan","key":"JIRAUSER10111","emailAddress":"zhangzhihan@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10111&avatarId=12001","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10111&avatarId=12001","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10111&avatarId=12001","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10111&avatarId=12001"},"displayName":"张智涵","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-02-06T13:48:03.334+0800","updated":"2023-02-06T13:48:03.334+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/32302/comment/53171","id":"53171","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=yangwei","name":"yangwei","key":"JIRAUSER10103","emailAddress":"yangwei@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10103&avatarId=10708","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10103&avatarId=10708","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10103&avatarId=10708","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10103&avatarId=10708"},"displayName":"杨威","active":true,"timeZone":"Asia/Shanghai"},"body":" 观察发现存在其他线程抢占收包线程时间片的情况，尝试给sapp绑的核（收包线程所在核）设置isolate，看看丢包现象是否有所改善","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=yangwei","name":"yangwei","key":"JIRAUSER10103","emailAddress":"yangwei@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10103&avatarId=10708","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10103&avatarId=10708","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10103&avatarId=10708","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10103&avatarId=10708"},"displayName":"杨威","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-02-07T09:38:01.963+0800","updated":"2023-02-07T09:38:01.963+0800"},{"self":"https://jira.geedge.net/rest/api/2/issue/32302/comment/54123","id":"54123","author":{"self":"https://jira.geedge.net/rest/api/2/user?username=zhangzhihan","name":"zhangzhihan","key":"JIRAUSER10111","emailAddress":"zhangzhihan@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10111&avatarId=12001","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10111&avatarId=12001","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10111&avatarId=12001","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10111&avatarId=12001"},"displayName":"张智涵","active":true,"timeZone":"Asia/Shanghai"},"body":"调整isolate后sapp丢包明显改善；\r\n目前仍有3台sapp在晚17点至晚23点间大量丢包，经排查发现该时间段内服务器cpu爆满。通过分析安全日志发现在该时间段内有一条DNS监测策略命中率非常高，导致cpu压力过大，选择一台sapp尝试关闭dns插件后，cpu使用从100%下降到65%，无丢包现象","updateAuthor":{"self":"https://jira.geedge.net/rest/api/2/user?username=zhangzhihan","name":"zhangzhihan","key":"JIRAUSER10111","emailAddress":"zhangzhihan@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10111&avatarId=12001","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10111&avatarId=12001","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10111&avatarId=12001","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10111&avatarId=12001"},"displayName":"张智涵","active":true,"timeZone":"Asia/Shanghai"},"created":"2023-02-21T14:59:42.603+0800","updated":"2023-02-21T14:59:42.603+0800"}],"maxResults":4,"total":4,"startAt":0},"votes":{"self":"https://jira.geedge.net/rest/api/2/issue/OMPUB-790/votes","votes":0,"hasVoted":false},"worklog":{"startAt":0,"maxResults":20,"total":0,"worklogs":[]},"assignee":{"self":"https://jira.geedge.net/rest/api/2/user?username=yangwei","name":"yangwei","key":"JIRAUSER10103","emailAddress":"yangwei@geedgenetworks.com","avatarUrls":{"48x48":"https://jira.geedge.net/secure/useravatar?ownerId=JIRAUSER10103&avatarId=10708","24x24":"https://jira.geedge.net/secure/useravatar?size=small&ownerId=JIRAUSER10103&avatarId=10708","16x16":"https://jira.geedge.net/secure/useravatar?size=xsmall&ownerId=JIRAUSER10103&avatarId=10708","32x32":"https://jira.geedge.net/secure/useravatar?size=medium&ownerId=JIRAUSER10103&avatarId=10708"},"displayName":"杨威","active":true,"timeZone":"Asia/Shanghai"},"updated":"2023-03-22T20:19:12.318+0800","status":{"self":"https://jira.geedge.net/rest/api/2/status/10103","description":"这一问题被认为是完成, 这项决议是正确的。问题已关闭可以重新开放。","iconUrl":"https://jira.geedge.net/images/icons/statuses/generic.png","name":"已关闭","id":"10103","statusCategory":{"self":"https://jira.geedge.net/rest/api/2/statuscategory/3","id":3,"key":"done","colorName":"green","name":"完成"}}}}