<section data-tool="mdnice编辑器" data-website="https://www.mdnice.com" style="margin-bottom: 0px;padding-left: 10px;padding-right: 10px;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;font-family: Optima, &quot;Microsoft YaHei&quot;, PingFangSC-regular, serif;font-size: 16px;color: rgb(0, 0, 0);line-height: 1.5em;word-spacing: 0em;letter-spacing: 0em;word-break: break-word;text-align: left;"> <p data-tool="mdnice编辑器" style="color: rgb(89, 89, 89);font-size: 15px;line-height: 1.8em;letter-spacing: 0.04em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">在这篇文章中，结合笔者在工作中使用到 Mybatis Plus 的过程中遇到的<strong style="color: rgb(53, 179, 120);background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">空字符串丢失</strong>的问题，谈一谈对 Mybatis Plus 的一些看法。由于项目历史问题，使用到的 Mybatis Plus 的版本如下：</p> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><span style="display: block;background: url(&quot;https://mmbiz.qpic.cn/mmbiz_svg/ibhzWy4ibIEpDFmKJ5DZYUnvT0XLJnNuiawXp9uHKj1H6sVEf50VWOZNtk25IGTOdTFck3lYdmV6iawdBOmR01ANBr0TZDPxcdco/640?wx_fmt=svg&amp;from=appmsg&quot;) 10px 10px / 40px no-repeat rgb(40, 44, 52);height: 30px;width: 100%;margin-bottom: -7px;border-radius: 5px;"></span><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="line-height: 26px;">&lt;<span style="color: #e06c75;line-height: 26px;">dependency</span>&gt;</span><br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="line-height: 26px;">&lt;<span style="color: #e06c75;line-height: 26px;">groupId</span>&gt;</span>com.baomidou<span style="line-height: 26px;">&lt;/<span style="color: #e06c75;line-height: 26px;">groupId</span>&gt;</span><br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="line-height: 26px;">&lt;<span style="color: #e06c75;line-height: 26px;">artifactId</span>&gt;</span>mybatis-plus-boot-starter<span style="line-height: 26px;">&lt;/<span style="color: #e06c75;line-height: 26px;">artifactId</span>&gt;</span><br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="line-height: 26px;">&lt;<span style="color: #e06c75;line-height: 26px;">version</span>&gt;</span>3.0-beta<span style="line-height: 26px;">&lt;/<span style="color: #e06c75;line-height: 26px;">version</span>&gt;</span><br><span style="line-height: 26px;">&lt;/<span style="color: #e06c75;line-height: 26px;">dependency</span>&gt;</span><br></code></pre> <h2 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;display: flex;"><span style="display: none;"></span><span style="font-size: 22px;color: rgb(53, 179, 120);line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">一、问题背景</span><span style="display: none;"></span></h2> <p data-tool="mdnice编辑器" style="color: rgb(89, 89, 89);font-size: 15px;line-height: 1.8em;letter-spacing: 0.04em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">我们存在如下数据库表：</p> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><span style="display: block;background: url(&quot;https://mmbiz.qpic.cn/mmbiz_svg/ibhzWy4ibIEpDFmKJ5DZYUnvT0XLJnNuiawXp9uHKj1H6sVEf50VWOZNtk25IGTOdTFck3lYdmV6iawdBOmR01ANBr0TZDPxcdco/640?wx_fmt=svg&amp;from=appmsg&quot;) 10px 10px / 40px no-repeat rgb(40, 44, 52);height: 30px;width: 100%;margin-bottom: -7px;border-radius: 5px;"></span><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">create</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">table</span>&nbsp;ad_person_send_rule<br>(<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">id</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #e6c07b;line-height: 26px;">bigint</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">unsigned</span>&nbsp;auto_increment<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;primary&nbsp;<span style="color: #c678dd;line-height: 26px;">key</span>,<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">status</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #e6c07b;line-height: 26px;">tinyint</span>(<span style="color: #d19a66;line-height: 26px;">3</span>)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">default</span>&nbsp;<span style="color: #d19a66;line-height: 26px;">1</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">not</span>&nbsp;<span style="color: #56b6c2;line-height: 26px;">null</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">comment</span>&nbsp;<span style="color: #98c379;line-height: 26px;">'状态'</span>,<br>&nbsp;&nbsp;&nbsp;&nbsp;operator_user&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #e6c07b;line-height: 26px;">varchar</span>(<span style="color: #d19a66;line-height: 26px;">128</span>)&nbsp;<span style="color: #c678dd;line-height: 26px;">charset</span>&nbsp;utf8&nbsp;<span style="color: #c678dd;line-height: 26px;">default</span>&nbsp;<span style="color: #98c379;line-height: 26px;">''</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">not</span>&nbsp;<span style="color: #56b6c2;line-height: 26px;">null</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">comment</span>&nbsp;<span style="color: #98c379;line-height: 26px;">'更新人'</span>,<br>&nbsp;&nbsp;&nbsp;&nbsp;account_type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #e6c07b;line-height: 26px;">varchar</span>(<span style="color: #d19a66;line-height: 26px;">10</span>)&nbsp;<span style="color: #c678dd;line-height: 26px;">charset</span>&nbsp;utf8&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">default</span>&nbsp;<span style="color: #98c379;line-height: 26px;">''</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">not</span>&nbsp;<span style="color: #56b6c2;line-height: 26px;">null</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">comment</span>&nbsp;<span style="color: #98c379;line-height: 26px;">'账户类型'</span>,<br>&nbsp;&nbsp;&nbsp;&nbsp;account_kinds&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #e6c07b;line-height: 26px;">varchar</span>(<span style="color: #d19a66;line-height: 26px;">10</span>)&nbsp;<span style="color: #c678dd;line-height: 26px;">charset</span>&nbsp;utf8&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">default</span>&nbsp;<span style="color: #98c379;line-height: 26px;">''</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">not</span>&nbsp;<span style="color: #56b6c2;line-height: 26px;">null</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">comment</span>&nbsp;<span style="color: #98c379;line-height: 26px;">'账户性质'</span>,<br>&nbsp;&nbsp;&nbsp;&nbsp;include_account_ids&nbsp;<span style="color: #e6c07b;line-height: 26px;">text</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">charset</span>&nbsp;utf8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #56b6c2;line-height: 26px;">null</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">comment</span>&nbsp;<span style="color: #98c379;line-height: 26px;">'允许的账户id'</span>,<br>&nbsp;&nbsp;&nbsp;&nbsp;exclude_account_ids&nbsp;<span style="color: #e6c07b;line-height: 26px;">text</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">charset</span>&nbsp;utf8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #56b6c2;line-height: 26px;">null</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">comment</span>&nbsp;<span style="color: #98c379;line-height: 26px;">'排除账户id'</span>,<br>&nbsp;&nbsp;&nbsp;&nbsp;create_time&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #e6c07b;line-height: 26px;">timestamp</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">default</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">CURRENT_TIMESTAMP</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">not</span>&nbsp;<span style="color: #56b6c2;line-height: 26px;">null</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">comment</span>&nbsp;<span style="color: #98c379;line-height: 26px;">'创建时间'</span>,<br>&nbsp;&nbsp;&nbsp;&nbsp;update_time&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #e6c07b;line-height: 26px;">timestamp</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">default</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">CURRENT_TIMESTAMP</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">not</span>&nbsp;<span style="color: #56b6c2;line-height: 26px;">null</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">on</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">update</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">CURRENT_TIMESTAMP</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">comment</span>&nbsp;<span style="color: #98c379;line-height: 26px;">'更新时间'</span><br>)<br></code></pre> <p data-tool="mdnice编辑器" style="color: rgb(89, 89, 89);font-size: 15px;line-height: 1.8em;letter-spacing: 0.04em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">对于 text 类型的 <code style="color: rgb(30, 107, 184);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;">exclude_account_ids</code> 字段，是不能设置默认值的，这是由于 MySQL 5.7 的严格模式导致的，什么都不写的话只能为空。对于业务来说，<code style="color: rgb(30, 107, 184);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;">exclude_account_ids</code> 空字符串是有意义的，所以在某些情况下，我们需要直接插入空字符串。</p> <h2 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;display: flex;"><span style="display: none;"></span><span style="font-size: 22px;color: rgb(53, 179, 120);line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">二、Mybatis Plus 执行插入过程丢失空字符串</span><span style="display: none;"></span></h2> <p data-tool="mdnice编辑器" style="color: rgb(89, 89, 89);font-size: 15px;line-height: 1.8em;letter-spacing: 0.04em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">构建对应的数据库实体类：</p> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><span style="display: block;background: url(&quot;https://mmbiz.qpic.cn/mmbiz_svg/ibhzWy4ibIEpDFmKJ5DZYUnvT0XLJnNuiawXp9uHKj1H6sVEf50VWOZNtk25IGTOdTFck3lYdmV6iawdBOmR01ANBr0TZDPxcdco/640?wx_fmt=svg&amp;from=appmsg&quot;) 10px 10px / 40px no-repeat rgb(40, 44, 52);height: 30px;width: 100%;margin-bottom: -7px;border-radius: 5px;"></span><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span>&nbsp;<span style="color: #e6c07b;line-height: 26px;">AdPersonSendRule</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">implements</span>&nbsp;<span style="color: #e6c07b;line-height: 26px;">Serializable</span>&nbsp;</span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #61aeee;line-height: 26px;">@TableField</span>(exist&nbsp;=&nbsp;<span style="color: #c678dd;line-height: 26px;">false</span>)<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">static</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">final</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">long</span>&nbsp;serialVersionUID&nbsp;=&nbsp;-<span style="color: #d19a66;line-height: 26px;">3646240230891328158L</span>;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #61aeee;line-height: 26px;">@TableId</span>(type&nbsp;=&nbsp;IdType.AUTO)<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;Long&nbsp;id;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;Integer&nbsp;status;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;String&nbsp;operatorUser;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;String&nbsp;accountType;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;String&nbsp;accountKinds;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;String&nbsp;includeAccountIds;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;String&nbsp;excludeAccountIds;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;Date&nbsp;createTime;<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;Date&nbsp;updateTime;<br>}<br></code></pre> <p data-tool="mdnice编辑器" style="color: rgb(89, 89, 89);font-size: 15px;line-height: 1.8em;letter-spacing: 0.04em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">执行插入操作：</p> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><span style="display: block;background: url(&quot;https://mmbiz.qpic.cn/mmbiz_svg/ibhzWy4ibIEpDFmKJ5DZYUnvT0XLJnNuiawXp9uHKj1H6sVEf50VWOZNtk25IGTOdTFck3lYdmV6iawdBOmR01ANBr0TZDPxcdco/640?wx_fmt=svg&amp;from=appmsg&quot;) 10px 10px / 40px no-repeat rgb(40, 44, 52);height: 30px;width: 100%;margin-bottom: -7px;border-radius: 5px;"></span><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">void</span>&nbsp;<span style="color: #61aeee;line-height: 26px;">createPersonAuditRuleTest</span><span style="line-height: 26px;">()</span>&nbsp;</span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;PersonAuditRuleRequest&nbsp;personAuditRuleRequest&nbsp;=&nbsp;<span style="color: #c678dd;line-height: 26px;">new</span>&nbsp;PersonAuditRuleRequest();<br>&nbsp;&nbsp;&nbsp;&nbsp;personAuditRuleRequest.setIncludeAccountIDs(<span style="color: #98c379;line-height: 26px;">"129,240,9123,100"</span>);<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #5c6370;font-style: italic;line-height: 26px;">//&nbsp;设置为空字符串</span><br>&nbsp;&nbsp;&nbsp;&nbsp;personAuditRuleRequest.setExcludeAccountIDs(<span style="color: #98c379;line-height: 26px;">""</span>);<br>&nbsp;&nbsp;&nbsp;&nbsp;personAuditRuleRequest.setAccountKinds(<span style="color: #c678dd;line-height: 26px;">new</span>&nbsp;HashSet&lt;&gt;());<br>&nbsp;&nbsp;&nbsp;&nbsp;personAuditRuleRequest.setAccountTypes(<span style="color: #c678dd;line-height: 26px;">new</span>&nbsp;HashSet&lt;&gt;());<br>&nbsp;&nbsp;&nbsp;&nbsp;Result&nbsp;robotAuditRule&nbsp;=&nbsp;personAuditController.createPersonAuditRule(personAuditRuleRequest);<br>&nbsp;&nbsp;&nbsp;&nbsp;log.info(<span style="color: #98c379;line-height: 26px;">"插入结果结果为：{}"</span>,&nbsp;JSONUtil.toJsonStr(robotAuditRule));<br>}<br></code></pre> <p data-tool="mdnice编辑器" style="color: rgb(89, 89, 89);font-size: 15px;line-height: 1.8em;letter-spacing: 0.04em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">其中我们的 excludeAccountIds 字段的值为空字符串，数据库的插入结果为：</p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100000378" data-ratio="0.13425925925925927" src="/upload/29eed4fd812a6122c537ef2ccc7699fe.png" data-type="png" data-w="1080" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="color: rgb(89, 89, 89);font-size: 15px;line-height: 1.8em;letter-spacing: 0.04em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">若把 excludeAccountIds 字段上标注 <code style="color: rgb(30, 107, 184);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;">@TableField</code> 注解后，数据库中插入结果为：</p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100000379" data-ratio="0.13425925925925927" src="/upload/bc1ec7cf271d11296522532567f16fce.png" data-type="png" data-w="1080" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="color: rgb(89, 89, 89);font-size: 15px;line-height: 1.8em;letter-spacing: 0.04em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">可见，没有带 <code style="color: rgb(30, 107, 184);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;">@TableField</code> 注解时，插入空字符串对应的值为 null，而带了 <code style="color: rgb(30, 107, 184);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;">@TableField</code> 注解时，插入空字符串对应的值为空字符串，这才是我想要的结果，但是，为什么会造成�

<p data-mpa-powered-by="yiban.io"><img src="/upload/08a263ede13d33448d78713510cc9d81.png" class="rich_pages wxw-img" data-ratio="0.125" data-w="640" data-imgfileid="100076887"></p> <section data-tool="mdnice编辑器" data-website="https://www.mdnice.com" style="margin-bottom: 0px;padding-left: 10px;padding-right: 10px;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;font-family: Optima, &quot;Microsoft YaHei&quot;, PingFangSC-regular, serif;font-size: 16px;color: rgb(0, 0, 0);line-height: 1.5em;word-spacing: 0em;letter-spacing: 0em;word-break: break-word;text-align: left;"> <h2 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="display: none;"></span><span style="font-size: 20px;color: rgb(0, 150, 136);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none solid;border-width: 1px 1px 1px 3px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">一、背景</span><span style="display: none;"></span></h2> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">某天，张三打算操作数据库，删除自己项目的无用数据，但是一不小心数据删多了。被误删的数据，如何恢复呢？本文将介绍相关方法，以及现成的一些工具。</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">例子：</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">有一个表</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #f92672;font-weight: bold;line-height: 26px;">create</span>&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">table</span>&nbsp;person<br>(<br>&nbsp;&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">id</span>&nbsp;&nbsp;&nbsp;<span style="color: #a6e22e;line-height: 26px;">bigint</span>&nbsp;primary&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">key</span>&nbsp;auto_increment&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">comment</span>&nbsp;<span style="color: #a6e22e;line-height: 26px;">'id'</span>,<br>&nbsp;&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">name</span>&nbsp;<span style="color: #a6e22e;line-height: 26px;">varchar</span>(<span style="line-height: 26px;">50</span>)&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">comment</span>&nbsp;<span style="color: #a6e22e;line-height: 26px;">'名称'</span><br>)&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">engine</span>&nbsp;=&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">innodb</span>;<br></code></pre> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">原本是要执行这条SQL语句：</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #f92672;font-weight: bold;line-height: 26px;">delete</span>&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">from</span>&nbsp;person&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">where</span>&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">id</span>&nbsp;&gt;&nbsp;<span style="line-height: 26px;">500000</span>;<br></code></pre> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">不小心执行了这条SQL语句：</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #f92672;font-weight: bold;line-height: 26px;">delete</span>&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">from</span>&nbsp;person;<br></code></pre> <h2 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="display: none;"></span><span style="font-size: 20px;color: rgb(0, 150, 136);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none solid;border-width: 1px 1px 1px 3px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">二、解决方案</span><span style="display: none;"></span></h2> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">处理这个问题的解决思路就是，基于binlog找回被删除的数据，将被删除的数据重新插入到数据库。</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">对于binlog文件来说，实际上保存的是对于数据库的正向操作。比如说，插入数据insert，binlog中保存的也是insert语句；删除数据delete，binlog中保存的也是delete语句。</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">因此，想要恢复被删除的数据，主要有两种方式：</p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100076885" data-ratio="0.43386243386243384" src="/upload/2b90a9edeaf25a7624d8cf7cbd987632.png" data-type="png" data-w="756" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">下面就针对上面的两种方式，进行详细的讲解</p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="display: none;"></span><span style="font-size: 18px;color: rgb(34, 34, 34);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none solid;border-width: 1px 1px 1px 2px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">1. 通用操作</span><span style="display: none;"></span></h3> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">首先介绍两种方式都需要使用到的一些通用的操作，主要用于设置binlog、找到binlog文件</p> <h5 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"></span><span style="color: rgb(34, 34, 34);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none dashed;border-width: 1px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;color: rgb(0, 0, 0);display: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span>1.1 确认binlog开启<span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;color: rgb(0, 0, 0);display: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span></span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"></span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span></h5> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><strong style="color: rgb(0, 150, 136);background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">1.1.1 查询开启状态</strong></p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">首先要保证binlog是开启的，不然数据肯定是没办法恢复回来的。</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">在MySQL中，可以通过执行以下SQL查询来检查是否已经开启了binlog：</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #f92672;font-weight: bold;line-height: 26px;">SHOW</span>&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">VARIABLES</span>&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">LIKE</span>&nbsp;<span style="color: #a6e22e;line-height: 26px;">'log_bin'</span>;<br></code></pre> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">这个查询将返回一个结果集，其中包含名为<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: &quot;Operator Mono&quot;, Consolas, Monaco, Menlo, monospace;word-break: break-all;">log_bin</code>的系统变量的值。如果<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: &quot;Operator Mono&quot;, Consolas, Monaco, Menlo, monospace;word-break: break-all;">log_bin</code>的值为ON，则表示binlog已经开启；如果值为OFF，则表示binlog没有开启。</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;">mysql&gt;&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">SHOW</span>&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">VARIABLES</span>&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">LIKE</span>&nbsp;<span style="color: #a6e22e;line-height: 26px;">'log_bin'</span>;<br>+<span style="color: #75715e;line-height: 26px;">---------------+-------+</span><br>|&nbsp;Variable_name&nbsp;|&nbsp;Value&nbsp;|<br>+<span style="color: #75715e;line-height: 26px;">---------------+-------+</span><br>|&nbsp;log_bin&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|&nbsp;ON&nbsp;&nbsp;&nbsp;&nbsp;|<br>+<span style="color: #75715e;line-height: 26px;">---------------+-------+</span><br>1&nbsp;row&nbsp;in&nbsp;<span style="color: #f92672;font-weight: bold;line-height: 26px;">set</span>&nbsp;(<span style="line-height: 26px;">0.01</span>&nbsp;sec)<br></code></pre> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><strong style="color: rgb(0, 150, 136);background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">1.1.2 开启binlog</strong></p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">如果发现没有开启，可以通过修改MySQL配置文件（通常是<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: &quot;Operator Mono&quot;, Consolas, Monaco, Menlo, monospace;word-break: break-all;">my.cnf</code>或<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: &quot;Operator Mono&quot;, Consolas, Monaco, Menlo, monospace;word-break: break-all;">my.ini</code>，Linux下MySQL的配置文件目录一般是<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: &quot;Operator Mono&quot;, Consolas, Monaco, Menlo, monospace;word-break: break-all;">/etc/mysql</code>）中的[<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: &quot;Operator Mono&quot;, Consolas, Monaco, Menlo, monospace;word-break: break-all;">mysqld</code>]部分来开启binlog。如果在配置文件中找到了类似以下的设置，则表示binlog已经开启：</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;">[mysqld]<br>log-bin=mysql-bin<br>server-id=1<br></code></pre> <ul data-tool="mdnice编辑器" style="list-style-type: square;margin-top: auto;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);font-size: 15px;line-height: 1.8em;letter-spacing: 0.06em;"> <p style="color: rgb(0, 0, 0);line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">修改配置启用了binlog之后，需要重启MySQL服务才能使更改生效</p> </section></li> <li> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);font-size: 15px;line-height: 1.8em;letter-spacing: 0.06em;"> <p style="color: rgb(0, 0, 0);line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;height: auto;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: &quot;Operator Mono&quot;, Consolas, Monaco, Menlo, monospace;word-break: break-all;">mysql-bin</code>表示binlog文件的前缀</p> </section></li> <li> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);font-size: 15px;line-height: 1.8em;letter-spacing: 0.06em;"> <p style="color: rgb(0, 0, 0);line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;height: auto;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: &quot;Operator Mono&quot;, Consolas, Monaco, Menlo, monospace;word-break: break-all;">server-id</code> 设置了MySQL服务器的唯一ID，必须设置ID，否则没办法开启binlog</p> </section></li> </ul> <h5 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"></span><span style="color: rgb(34, 34, 34);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none dashed;border-width: 1px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;color: rgb(0, 0, 0);display: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span>1.2 binlog模式<span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;color: rgb(0, 0, 0);display: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span></span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: no

## 导入所需的jar包 ``` <dependency> <groupId>org.bouncycastle</groupId> <artifactId>bcprov-jdk15on</artifactId> <version>1.69</version> </dependency> <dependency> <groupId>org.bouncycastle</groupId> <artifactId>bcpkix-jdk15on</artifactId> <version>1.69</version> <!-- 同样，请根据需要替换为最新版本 --> </dependency> ``` ## 生成证书的代码 ``` package cn.studyjava; import java.io.FileWriter; import java.io.IOException; import java.security.KeyPair; import java.security.KeyPairGenerator; import java.security.NoSuchAlgorithmException; import java.security.PrivateKey; import java.security.PublicKey; import java.security.Security; import org.bouncycastle.jce.provider.BouncyCastleProvider; import org.bouncycastle.openssl.PEMWriter; public class StudyJavaApplication { static { // 添加 BouncyCastle 提供者 Security.addProvider(new BouncyCastleProvider()); } public static void main(String[] args) throws NoSuchAlgorithmException, IOException { // 1. 创建一个 RSA 密钥对生成器 KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("RSA"); keyPairGenerator.initialize(2048); // 密钥位数，可以选择 2048 或更高 // 2. 生成密钥对 KeyPair keyPair = keyPairGenerator.generateKeyPair(); PublicKey publicKey = keyPair.getPublic(); PrivateKey privateKey = keyPair.getPrivate(); // 3. 将公钥和私钥写入文件 writePemFile("public_key.pem", publicKey); writePemFile("private_key.pem", privateKey); } // 写入 PEM 文件 private static void writePemFile(String fileName, Object key) throws IOException { try (PEMWriter pemWriter = new PEMWriter(new FileWriter(fileName))) { pemWriter.writeObject(key); } } } ``` 到此就可以生成公私钥了。

大家好，我是大厂面试喜欢问到的redis，看完视频你会对我有一个全新的了解！！！

我叫你哥们儿，你却说我是男的！！！

<section style="margin-bottom: 0px;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-imgfileid="100036198" data-ratio="0.2222222222222222" src="/upload/e6027109404f4c4053d14e8284b886e9.png" data-w="1080" style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: var(--articleFontsize);letter-spacing: 0.034em;"> <br> </section> <section style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);visibility: visible;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="color: rgb(178, 178, 178);font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 12px;letter-spacing: 1px;background-color: rgb(255, 255, 255);">本文将基于抖音集团内部两大业务的典型实时数仓场景，介绍Paimon在抖音集团内部的生产实践。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="246" data-backw="578" data-galleryid="" data-imgfileid="100036200" data-ratio="0.42592592592592593" data-s="300,640" src="/upload/4f674f3652c52532ab53344e3b5b4056.png" data-type="png" data-w="1080" style="width: 100%;height: auto;"> </section> <section style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);text-align: right;visibility: visible;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;">作者：李明、苏兴、文杰</strong></span> </section> <h1 style="-webkit-tap-highlight-color: transparent;margin-right: 8px;margin-left: 8px;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);text-align: right;visibility: visible;"></h1> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;">抖音集团大数据工程师</strong></span> <br> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <hr style="border-style: solid;border-width: 1px 0 0;border-color: rgba(0,0,0,0.1);-webkit-transform-origin: 0 0;-webkit-transform: scale(1, 0.5);transform-origin: 0 0;transform: scale(1, 0.5);"> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"></strong></span> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;">目前抖音集团内部主要使用 Lambda 架构进行实时数仓建设，其中实时处理链路主要采用 Flink + MQ 进行实现。</span> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;">在 Lambda 架构体系下，主要优势是数据新鲜度高，但采用两条处理链路也带来了其它问题：</span> <br> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">1. 维护成本高：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;">需要维护实时、离线两条不同技术栈的处理链路，开发和维护成本高；</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">2. 计算口径难对齐：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;">没有统一的 Table 抽象，Schema 难对齐；两条链路同时跑，计算语义难对齐；</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">3. OLAP 查询能力差：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;">消息队列只支持 APPEND 流，从流式数据转化为 Table 的成本开销高，需要不断处理 changelog，导致 OLAP 查询能力差；</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">4. 问题排查困难：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;">如果数据有问题，用户需要排查数据 Pipeline，但由于中间结果不可查，导致排查难度高；</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">5. 数据订正困难：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;">实时链路数据订正需要大量的人工介入，修改逻辑、双跑等，数据订正困难。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">随着流式计算引擎的不断完善，以 Flink 为代表的流式计算引擎提出了新的目标：</span> <strong><span style="font-size: 15px;letter-spacing: 1px;color: rgb(0, 128, 255);">为有限数据和无限数据提供一套统一的处理 API。</span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">流批一体的计算模式进一步简化了数仓生产体系，将计算链路收敛到了相同的技术栈，降低了开发和维护成本。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">同时随着数据湖技术的兴起，它能够支持高效的数据流 / 批读写、数据回溯以及数据更新，进一步解决了Lambda 架构体系下的其它问题。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">Apache Paimon是一项流式数据湖存储技术，基于Flink Table Store独立孵化出来的项目，<strong><span style="font-size: 15px;letter-spacing: 1px;color: rgb(0, 128, 255);">主要目标是解决流式场景中的常见问题，为用户提供高吞吐、低延迟的数据摄入、流式订阅和实时查询，</span></strong>支持主流的计算 / OLAP 引擎，尤其对 Flink 的支持最佳。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">因此在经过调研后，最终决定采用 Apache Paimon 作为数据湖底座，和业务进行新一代实时数仓建设。</span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"></span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">本文将基于抖音集团两个业务的典型实时数仓场景，介绍 Apache Paimon 在抖音集团内部的生产实践。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section data-mpa-template="t" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);visibility: visible;"> <section data-mpa-template="t" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"> <section data-mpa-category="模板" data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;"> <section style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;margin-left: 8px;margin-right: 8px;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;display: flex;align-items: flex-end;justify-content: space-between;border-bottom: 1px solid rgba(9, 9, 9, 0.55);visibility: visible;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;border-bottom: 6px solid rgb(0, 110, 255);visibility: visible;"> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;visibility: visible;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;Helvetica Neue&quot;, Helvetica, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;visibility: visible;">01</span></p> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 26px;letter-spacing: 0.034em;">场景一：游戏视频指标上卷</span></p> </section> </section> </section> </section> </section> </section> <section style="-webkit-tap-highlight-color: transparent;margin: 8px 8px 0px;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"> <br> </section> <section style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 业务场景</span></strong> </section> <section style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">游戏-新游场景在公测宣发、测试上线首日、首发等相关节点，产品和运营需要根据游戏短视频的点赞、曝光、评论等实时指标在第一时间挖掘优质作者和发现潜力热点。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">游戏实时数仓团队当前通过接入短视频实时数仓团队的分钟粒度流并关联游戏相关维表，通过分钟粒度上卷到天粒度指标的方案来提供相关指标。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 原有方案</span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="162" data-backw="578" data-galleryid="" data-imgfileid="100036201" data-ratio="0.2796296296296296" data-s="300,640" src="/upload/8f073af6a895d40d61dbc62fa34c41d5.png" data-type="png" data-w="1080" style="width: 100%;height: auto;"> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"></span></strong> </section> <section style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><strong style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 方案痛点</span></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><strong style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">1. 由于短视频 topic 流量在 100w+/s 左右，即使 Lookup Join HitRate 平均在 90% 左右，但是全链路峰值仍有60w+/s 的流量打到维表存储，<strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;color: rgb(0, 128, 255);">给维表服务带来比较大的访问压力。</span></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="89" data-backw="578" data-height="534" data-imgfileid="100036203" data-ratio="0.1537037037037037" src="/upload/936ef7572e992859cf2a857fd6548e9f.png" data-type="png" data-w="1080" data-width="3482" style="width: 100%;height: auto;"> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);">Lookup Join HitRate</span></em></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <em><span style="color: rgb(143, 149, 158);font-size: 14px;"><br></span></em> </section> <section style="margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-height="538" data-imgfileid="100036202" data-ratio="0.15092592592592594" src="/upload/4cddf7ab18a154341ca83689ba12425a.png" data-type="png" data-w="1080" data-width="3560"> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);">Lookup Join Request Per Second&nbsp;</span></em></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);"><br></span></em></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">2. 由于上卷任务的 source 是 append 流，分钟粒度的指标会实时的变化，所以需要消费 source 后通过 MAX / LAST_VALUE 等聚合函数去构建 retract 流、处理乱序等问题，<strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;color: rgb(0, 128, 255);">开发效率低且增加额外的状态成本。</span></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span data-lark-record-data="{&quot;rootId&quot;:&quot;Ms3tdWpIroo7Y0xkipzc5F7VnYd&quot;,&quot;text&quot;:{&quot;initialAttributedTexts&quot;:{&quot;text&quot;:{&quot;0&quot;:&quot;由于上卷任务的 source 是 append 流，分钟粒度的指标会实时的变化，所以需要消费 source 后通过 MAX / LAST_VALUE 等聚合函数去构建 retract 流、处理乱序等问题，开发效率低且增加额外的状态成本。&quot;},&quot;attribs&quot;:{&quot;0&quot;:&quot;*0+3a&quot;}},&quot;apool&quot;:{&quot;numToAttrib&quot;:{&quot;0&quot;:[&quot;author&quot;,&quot;6891832921678954498&quot;]},&quot;nextNum&quot;:1}},&quot;type&quot;:&quot;ordered&quot;,&quot;referenceRecordMap&quot;:{},&quot;extra&quot;:{&quot;channel&quot;:&quot;saas&quot;,&quot;pasteRandomId&quot;:&quot;ab0c606d-57dd-49d4-8c87-a553fcf2c0b1&quot;,&quot;mention_page_title&quot;:{},&quot;external_mention_url&quot;:{}},&quot;isKeepQuoteContainer&quot;:false,&quot;isFromCode&quot;:false,&quot;selection&quot;:[{&quot;id&quot;:23,&quot;type&quot;:&quot;text&quot;,&quot;selection&quot;:{&quot;start&quot;:0,&quot;end&quot;:118},&quot;recordId&quot;:&quot;doxcnJmyG2OGJu9Rsdw4apWT2Pg&quot;}],&quot;payloadMap&quot;:{},&quot;isCut&quot;:false}" data-lark-record-format="docx/text"></span> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">CREATE</span> <span class="code-snippet__keyword">VIEW</span> view_01 <span class="code-snippet__keyword">AS</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">SELECT</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f3) <span class="code-snippet__keyword">AS</span> f3,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> f5,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f6) <span class="code-snippet__keyword">AS</span> f6,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f7) <span class="code-snippet__keyword">AS</span> f7,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">LAST_VALUE</span>(f8, f5) <span class="code-snippet__keyword">AS</span> f8</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">FROM</span> source_table</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">GROUP</span> <span class="code-snippet__keyword">BY</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> f5;</span></code> <code><span class="code-snippet_outer"><br></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">INSERT</span> <span class="code-snippet__keyword">INTO</span> sink_table</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">SELECT</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">SUM</span>(f3) <span class="code-snippet__keyword">AS</span> f3,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">CAST</span>(f2 <span class="code-snippet__keyword">AS</span> <span class="code-snippet__built_in">BIGINT</span>) <span class="code-snippet__keyword">AS</span> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f4) <span class="code-snippet__keyword">AS</span> f4,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f5) <span class="code-snippet__keyword">AS</span> f5,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f6) <span class="code-snippet__keyword">AS</span> f6,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f7) <span class="code-snippet__keyword">AS</span> f7,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">LAST_VALUE</span>(f8, f5) <span class="code-snippet__keyword">AS</span> f8</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">FROM</span> view_01</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">GROUP</span> <span class="code-snippet__keyword">BY</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2;</span></code> </section></pre> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);"><br></span></em></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);"><br></span></em></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-size: 15px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/&nbsp;Paimon实践</span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-size: 15px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">使用 Paimon 作为游戏维表，在 Flink 中 Lookup Join 将打宽结果写入 Paimon 表中，Paimon 表基于 lookup changelog producer 产生完整的 changelog，下游消费 changelog 做上卷计算。在存储层基于 Paimon 的 Sequence Field 能力处理乱序。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="165" data-backw="578" data-galleryid="" data-imgfileid="100036204" data-ratio="0.2861111111111111" data-s="300,640" src="/upload/6a9f1b75d15431a6719a4bd6bde6f8be.png" data-type="png" data-w="1080" style="width: 100%;height: auto;"> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">1. 维表打宽</span></strong> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__comment">--维表模型DDL</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">create</span> <span class="code-snippet__keyword">table</span> dim_table01 (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`id`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f1`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f2`</span> <span class="code-snippet__built_in">BIGINT</span></span></code> <code><span class="code-snippet_outer"> PRIMART <span class="code-snippet__keyword">KEY</span> (f1) <span class="code-snippet__keyword">NOT</span> <span class="code-snippet__keyword">ENFORCED</span></span></code> <code><span class="code-snippet_outer">) <span class="code-snippet__keyword">WITH</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer'</span>=<span class="code-snippet__string">'lookup'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer.row-deduplicate'</span>=<span class="code-snippet__string">'true'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'sequence.field'</span>=<span class="code-snippet__string">'f2'</span>,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer">)</span></code> <code><span class="code-snippet_outer"> </span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">create</span> <span class="code-snippet__keyword">table</span> dim_table02 (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`id`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f1`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f2`</span> <span class="code-snippet__built_in">BIGINT</span></span></code> <code><span class="code-snippet_outer"> PRIMART <span class="code-snippet__keyword">KEY</span> (f1) <span class="code-snippet__keyword">NOT</span> <span class="code-snippet__keyword">ENFORCED</span></span></code> <code><span class="code-snippet_outer">) <span class="code-snippet__keyword">WITH</span>(</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer'</span>=<span class="code-snippet__string">'lookup'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer.row-deduplicate'</span>=<span class="code-snippet__string">'true'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'sequence.field'</span>=<span class="code-snippet__string">'f2'</span>,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer">)</span></code> <code><span class="code-snippet_outer"><br></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__comment">--分钟指标流关联维度</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">SELECT</span></span></code> <code><span class="code-snippet_outer"> AA.id,</span></code> <code><span class="code-snippet_outer"> BB.f1 <span class="code-snippet__keyword">as</span> bb_f1,</span></code> <code><span class="code-snippet_outer"> CC.f1 <span class="code-snippet__keyword">as</span> cc_f1</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">FROM</span> source_table AA</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">LEFT</span> <span class="code-snippet__keyword">JOIN</span></span></code> <code><span class="code-snippet_outer"> paimon.db_name.dim_table01 <span class="code-snippet__comment">/*+ OPTIONS('lookup.async'='true', 'lookup.async-thread-number'='8') */</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">FOR</span> SYSTEM_TIME <span class="code-snippet__keyword">AS</span> <span class="code-snippet__keyword">OF</span> proctime <span class="code-snippet__keyword">AS</span> BB</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">ON</span> AA.id = BB.id</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">LEFT</span> <span class="code-snippet__keyword">JOIN</span></span></code> <code><span class="code-snippet_outer"> paimon.db_name.dim_table02 <span class="code-snippet__comment">/*+ OPTIONS('lookup.async'='true', 'lookup.async-thread-number'='8') */</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">FOR</span> SYSTEM_TIME <span class="code-snippet__keyword">AS</span> <span class="code-snippet__keyword">OF</span> proctime <span class="code-snippet__keyword">AS</span> CC</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">ON</span> AA.id = CC.id;</span></code> </section></pre> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(0, 0, 0);visibility: visible;font-size: 15px;letter-spacing: 1px;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(0, 0, 0);visibility: visible;font-size: 15px;letter-spacing: 1px;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;">2. 指标上卷</strong></span> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__comment">--分钟指标模型DDL</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">create</span> <span class="code-snippet__keyword">table</span> <span class="code-snippet__string">`db_name`</span>.<span class="code-snippet__string">`table_name`</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`id`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f1`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f2`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f3`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f4`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f5`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f6`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f7`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f8`</span> <span class="code-snippet__keyword">map</span>&lt;<span class="code-snippet__keyword">STRING</span>, <span class="code-snippet__keyword">STRING</span>&gt;</span></code> <code><span class="code-snippet_outer"> PRIMARY <span class="code-snippet__keyword">KEY</span> (<span class="code-snippet__keyword">id</span>, f1, f2, f4, f5, <span class="code-snippet__built_in">date</span>, <span class="code-snippet__keyword">hour</span>) <span class="code-snippet__keyword">NOT</span> <span class="code-snippet__keyword">ENFORCED</span></span></code> <code><span class="code-snippet_outer">) PARTITIONED <span class="code-snippet__keyword">by</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`date`</span> <span class="code-snippet__keyword">STRING</span> <span class="code-snippet__keyword">comment</span> <span class="code-snippet__string">'日期'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`hour`</span> <span class="code-snippet__keyword">STRING</span> <span class="code-snippet__keyword">comment</span> <span class="code-snippet__string">'小时'</span></span></code> <code><span class="code-snippet_outer">) <span class="code-snippet__keyword">WITH</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer'</span>=<span class="code-snippet__string">'lookup'</span>, </span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'partition.expiration-time'</span>=<span class="code-snippet__string">'30 d'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'partition.timestamp-pattern'</span>=<span class="code-snippet__string">'$date'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'partition.expiration-check-interval'</span>=<span class="code-snippet__string">'3h'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'sequence.field'</span>=<span class="code-snippet__string">'f8,f3'</span>,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer">);</span></code> <code><span class="code-snippet_outer"><br></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__comment">--分钟指标上卷</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">INSERT</span> <span class="code-snippet__keyword">INTO</span> sink_table</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">SELECT</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">SUM</span>(f3) <span class="code-snippet__keyword">AS</span> f3,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f4) <span class="code-snippet__keyword">AS</span> f4,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f5) <span class="code-snippet__keyword">AS</span> f5,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f6) <span class="code-snippet__keyword">AS</span> f6,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f7) <span class="code-snippet__keyword">AS</span> f7,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">LAST_VALUE</span>(f8, f7) <span class="code-snippet__keyword">AS</span> f8</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">FROM</span> paimon.db_name.table_name</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">GROUP</span> <span class="code-snippet__keyword">BY</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2;</span></code> </section></pre> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><strong style="font-size: 15px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/&nbsp;方案收益</span></strong></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">1. 流批一体模式开发：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">原有链路需要使用 MAX、LAST_VALUE 等函数来构造 retract 消息，以保证下游 SUM 计算结果正确，流与批的开发模式是分割的。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">基于 Paimon 存储数据并补齐 changelog，开发模式流与批是对齐的，获得流批一体的开发体验，提高了开发效率。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">2. 维表新鲜度更高：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">原有链路中为了减少维表服务压力，所以本地 Cache TTL设置为 50 min，数据新鲜度较低，在Paimon 维表中默认每 10s 会主动检查维表数据是否有更新，并主动更新本地缓存，数据新鲜度更高。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">3. 数据乱序问题：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">原链路中需要使用 LAST_VALUE 来处理数据乱序问题，增加了额外的状态成本。Paimon 合并数据时可以根据 sequence.field 来排序，从而在存储层解决数据乱序问题，不需要在 Flink 中维护状态。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <p style="text-align: center;"><img class="rich_pages wxw-img" data-galleryid="" data-imgfileid="100036220" data-ratio="0.42592592592592593" data-s="300,640" src="/upload/bf75fb0811044ae7354738d75dae2b99.png" data-type="png" data-w="1080" style=""></p> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section data-mpa-template="t" mpa-from-tpl="t" style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);visibility: visible;"> <section data-mpa-template="t" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"> <section data-mpa-category="模板" data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;"> <section style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;margin-left: 8px;margin-right: 8px;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;display: flex;align-items: flex-end;justify-content: space-between;border-bottom: 1px solid rgba(9, 9, 9, 0.55);visibility: visible;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;border-bottom: 6px solid rgb(0, 110, 255);visibility: visible;"> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;visibility: visible;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;Helvetica Neue&quot;, Helvetica, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;visibility: visible;">02</span></p> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 26px;letter-spacing: 0.034em;">场景二：财经多流拼接</span></p> </section> </section> </section> </section> </section> </section> <section style="margin: 8px 8px 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"> <br> </section> <section style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 业务场景</span></strong> </section> <section style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">由于前端业务过程是由于多个功能模块通过接口拼接而成的，历史链路无法打通，关键节点指标缺失，<strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">DA同学需要投入大量时间找数据拼凑数据，用数成本高、数据精度低、无法实时看到增长策略表现。</span></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">想要获得完整的用户交易过程的明细数据，财经侧在发起支付交易的时候创建了 trace_id，然后从支付开始透传到用户支付结束的所有流程中，利用 trace_id 构建用户交易行为宽表；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">宽表打宽的过程中，会去组合前后端埋点、会员、标签、策略、交易、营销等多个主题域的数据，降低业务同学用数、找数成本。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);">/ 原有业务方案</span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><br></span></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="272" data-backw="578" data-galleryid="" data-imgfileid="100036205" data-ratio="0.4703296703296703" data-s="300,640" src="/upload/ef317cebe62376e63518c5288e195203.png" data-type="png" data-w="910" style="width: 100%;height: auto;"> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">insert</span> <span class="code-snippet__keyword">into</span> sink</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">select</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">last_value</span>(f1) <span class="code-snippet__keyword">as</span> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">last_value</span>(f2) <span class="code-snippet__keyword">as</span> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">last_value</span>(f3) <span class="code-snippet__keyword">as</span> f3,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">last_value</span>(f4) <span class="code-snippet__keyword">as</span> f4,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">from</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">select</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f3,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f4,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">from</span> table1</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">union</span> <span class="code-snippet__keyword">all</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">select</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f2,</span></code> <code><span class="code-snippet_outer"> f3,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">from</span> table2</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">union</span> <span class="code-snippet__keyword">all</span></span></code> <code><span class="code-snippet_outer"> ......</span></code> <code><span class="code-snippet_outer"> )</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">group</span> <span class="code-snippet__keyword">by</span> <span class="code-snippet__keyword">id</span>;</span></code> </section></pre> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);">/ 方案痛点</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">1. 资源开销和运维成本高：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">当前基于 MQ 和 Flink 做打宽任务的多流 Join，状态大小超过了 10TB，计算资源 1600+ CU。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">大状态会导致资源开销变高、任务吞吐抖动、故障恢复时间变长等问题。在当前场景下，宽表任务的异常抖动会带来下游超 10 min 的断流感知，运维成本较高，亟需优化。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="213" data-backw="562" data-height="1170" data-imgfileid="100036206" data-ratio="0.37962962962962965" src="/upload/2c5d33b6c3354307b2f63e19824ec00e.png" data-type="png" data-w="1080" data-width="3080" style="width: 100%;height: auto;"> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="color: rgb(0, 128, 255);"><strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">2. 数据乱序问题：</span></strong></span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">由于打宽任务的超大状态，因此在 Flink 任务中状态的 TTL 配置相对较小（小时级）。在状态过期后，乱序数据会导致拼接结果不正确问题，产生额外的运维和排查成本</span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);">/ Paimon实践</span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><br></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">结合 Paimon 的 Partial Update 能力，对财经用户行为打宽任务进行了改写，数据直接写入到 Paimon 表，原链路中的聚合算子得以消除，任务状态大幅下降。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">同时基于 Paimon 的 Sequence Group 机制，为每个流定义了相应字段的顺序，避免因为乱序出现的数据不一致问题。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="249" data-backw="578" data-galleryid="" data-imgfileid="100036207" data-ratio="0.43037974683544306" data-s="300,640" src="/upload/9a58efb438c2d37a0fd1b28eb8bee6b3.png" data-type="png" data-w="948" style="width: 100%;height: auto;"> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">CREATE</span> <span class="code-snippet__keyword">TABLE</span> t (</span></code> <code><span class="code-snippet_outer"> trace_id <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> f1 <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> f2 <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> g_1 <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> f3 <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> f4 <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> g_2 <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> PRIMARY <span class="code-snippet__keyword">KEY</span> (trace_id) <span class="code-snippet__keyword">NOT</span> <span class="code-snippet__keyword">ENFORCED</span></span></code> <code><span class="code-snippet_outer">) <span class="code-snippet__keyword">WITH</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'merge-engine'</span>=<span class="code-snippet__string">'partial-update'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'fields.g_1.sequence-group'</span>=<span class="code-snippet__string">'f1,f2'</span>, <span class="code-snippet__comment">-- f1,f2字段根据 g_1 排序</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'fields.g_2.sequence-group'</span>=<span class="code-snippet__string">'f3,f4'</span> <span class="code-snippet__comment">-- f3,f4字段根据 g_2 排序</span></span></code> <code><span class="code-snippet_outer">);</span></code> <code><span class="code-snippet_outer"><br></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">insert</span> <span class="code-snippet__keyword">into</span> t</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">select</span> trace_id,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> g_1,</span></code> <code><span class="code-snippet_outer"> f3,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> g_2,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">from</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">select</span> trace_id,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> g_1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f3,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f4,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__built_in">BIGINT</span>) <span class="code-snippet__keyword">as</span> g_2,</span></code> <code><span class="code-snippet_outer"> xxx</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">from</span> table1</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">union</span> <span class="code-snippet__keyword">all</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">select</span> trace_id,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__built_in">BIGINT</span>) <span class="code-snippet__keyword">as</span> g_1,</span></code> <code><span class="code-snippet_outer"> f3,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> g_2,</span></code> <code><span class="code-snippet_outer"> xxx</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">from</span> table2</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">union</span> <span class="code-snippet__keyword">all</span></span></code> <code><span class="code-snippet_outer"> ......</span></code> <code><span class="code-snippet_outer"> )</span></code> </section></pre> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);">/ 方案收益</span><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"></span></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><span style="font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><br></span></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">1. 计算资源下降：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">计算资源优化 50%+ (1600 CU 缩减到 800 CU)，收益主要来源于状态管理成本下降；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">2. 状态优化：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">消除聚合状态后，作业状态由 10TB 缩减到小于 20GB；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">3. 开发和运维成本下降：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">中间数据可查可复用，同时指标增减可以通过 DDL 直接操作 Paimon 表；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;text-align: justify;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">4. 数据乱序问题解决成本低：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;text-align: justify;font-size: 15px;letter-spacing: 1px;">基于 Paimon 的 Sequence Group 机制可以对多流数据按序进行合并，处理更长时间范围内的乱序问题，并且不额外新增状态成本，较之于原链路方案，数据质量提升6%</span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: var(--articleFontsize);letter-spacing: 0.034em;text-align: justify;">。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <p style="text-align: center;"><img class="rich_pages wxw-img" data-galleryid="" data-imgfileid="100036221" data-ratio="0.44722222222222224" data-s="300,640" src="/upload/2630e23b908a6cc008b6194ae4c135f4.png" data-type="png" data-w="1080" style=""></p> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: var(--articleFontsize);letter-spacing: 0.034em;text-align: justify;"><br></span> </section> <section data-mpa-template="t" mpa-from-tpl="t" style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);visibility: visible;"> <section data-mpa-template="t" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"> <section data-mpa-category="模板" data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;"> <section style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;margin-left: 8px;margin-right: 8px;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;display: flex;align-items: flex-end;justify-content: space-between;border-bottom: 1px solid rgba(9, 9, 9, 0.55);visibility: visible;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;border-bottom: 6px solid rgb(0, 110, 255);visibility: visible;"> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;visibility: visible;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;Helvetica Neue&quot;, Helvetica, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;visibility: visible;">03</span></p> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 26px;letter-spacing: 0.034em;">未来展望</span></p> </section> </section> </section> </section> </section> </section> <section style="margin: 8px 8px 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: &quot;PingFang SC&quot;, system-ui, -apple-system, &quot;system-ui&quot;, &quot;Helvetica Neue&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">我们期望未来以 Flink 为核心，以数据湖为底座，为用户提供全链路数据生产和管理的实时数仓解决方案，进一步简化用户的开发和使用成本。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">我们也将继续针对实际业务场景进行 Apache Paimon 优化，包括：</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">1. 千列大宽表合并性能优化：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">LSM Tree 架构使得 Apache Paimon 有很高的点查与合并性能，但在超大列数的业务场景中性能下降较多，内部将针对这一场景进行优化；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">2. 维表性能优化：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">Apache Paimon 的本地维表可以极大的减少传统外部 KV 存储的请求数量，但在大流量场景中，我们注意到本地维表刷新是同步的，同时没有按照 bucket 进行 shuffle，导致维表变化较快时，吞吐有明显尖刺，我们将结合 Flink 继续优化维表的访问性能；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">3. Merge Engine 扩展：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei UI&quot;, &quot;Microsoft YaHei&quot;, Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">部分业务场景中，业务需要自定义的 Merge Engine 来实现更加复杂的合并策略，因此我们将扩展 Merge Engine，使其支持业务进行扩展以应对更加复杂的业务场景。</span> </section>

<p style="margin-top: 15px;margin-bottom: 15px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);">RabbitMQ本身并不直接提供防止消息重复消费的机制，但可以通过一系列的策略和措施来尽量避免或处理消息的重复消费。以下是一些常用的方法：</p> <p style="margin-top: 15px;margin-bottom: 15px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);"><img class="rich_pages wxw-img" data-imgfileid="100000679" data-ratio="0.74625" src="/upload/429dab78221ad0a34592aab358bac808.jpg" data-type="jpeg" data-w="1600" style="vertical-align: middle;border-width: 0px;border-style: initial;border-color: initial;"></p> <h3 style="margin-top: 25px;margin-bottom: 15px;font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;color: rgb(68, 68, 68);font-size: 18px;line-height: 24px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);"><strong>一、消息确认机制</strong></h3> <ol style="margin-top: 15px;margin-bottom: 15px;padding-left: 30px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);" class="list-paddingleft-1"> <li><p><span style="font-weight: 700;">自动确认模式</span>：在这种模式下，当消费者接收到消息后，RabbitMQ会自动将该消息标记为已确认，并从队列中删除。但这种方式存在风险，因为一旦消息被接收，无论处理成功与否，它都会被认为已经处理完毕。为了避免重复消费，可以谨慎使用这种模式，或结合其他机制进行补充。</p></li> <li><p><span style="font-weight: 700;">手动确认模式</span>：消费者处理完消息后，需要显式地向RabbitMQ发送一个确认消息（ACK），以告知RabbitMQ该消息已被正确处理。如果消费者在处理过程中发生异常或崩溃，可以选择不发送确认消息，这样RabbitMQ会将该消息重新投递给其他消费者。这种方式更加可靠，但需要消费者编写额外的代码来处理确认消息的发送。</p></li> </ol> <section> <mp-common-cpsad class="js_uneditable custom_select_card new_cps_iframe" data-pluginname="mpcps" data-adtype="short-play" data-playappid="wxbd2b3445f6af2282" data-planid="202411011728216291183" data-dramaid="212569" srcappid="wx690f86c2e4552666" data-templateid="video-play" data-defaultpath="plugin-private%3A%2F%2Fwx94a6522b1d640c3b%2Fpages%2Fplaylet%2Fplaylet%3FdramaId%3D212569%26srcAppid%3Dwx690f86c2e4552666%26wxTicket%3DdGlja2V0ODEwNjkwNTQwNzgyMzYzODc4NTMzNzg1OQ" data-goodssouce="1" data-traceid="3ea45637-221f-413d-a156-f4049eece0a4212569" data-videocarddata="{&quot;name&quot;:&quot;婚礼发癫后，奶狗总裁追着我宠&quot;,&quot;categoryName&quot;:&quot;都市/职场/爱情/家庭&quot;,&quot;videoCoverUrl&quot;:&quot;https://wxaintpcos.wxqcloud.qq.com.cn/public/wx690f86c2e4552666/WxaDramaCoverImage/66502949042c807fc775f10405901188.jpg&quot;,&quot;categoryInfo&quot;:&quot;80集&quot;}" data-mediano="1" data-disablechangevideo="1" data-playcpstype="2"></mp-common-cpsad> </section> <h3 style="margin-top: 25px;margin-bottom: 15px;font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;color: rgb(68, 68, 68);font-size: 18px;line-height: 24px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);"></h3> <h3 style="margin-top: 25px;margin-bottom: 15px;font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;color: rgb(68, 68, 68);font-size: 18px;line-height: 24px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);"><strong>二、幂等性设计</strong></h3> <p style="margin-top: 15px;margin-bottom: 15px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);">幂等性是指无论执行多少次操作，其结果都保持一致。在设计消费者处理逻辑时，确保无论接收到多少次相同的消息，处理结果都相同。这通常涉及到对业务逻辑的仔细设计，例如：</p> <ol style="margin-top: 15px;margin-bottom: 15px;padding-left: 30px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);" class="list-paddingleft-1"> <li><p><span style="font-weight: 700;">数据库操作</span>：在执行数据库插入、更新等操作时，使用唯一索引、条件更新等手段来防止重复执行某些操作。例如，数据库插入时，通过唯一索引避免重复插入同一条记录；数据库更新时，使用条件更新（如UPDATE WHERE）确保只在特定条件满足时更新，避免重复更新。</p></li> <li><p><span style="font-weight: 700;">业务逻辑</span>：在业务逻辑中加入状态检查或去重逻辑，确保即使消息被重复处理，系统状态也不会发生变化。</p></li> </ol> <h3 style="margin-top: 25px;margin-bottom: 15px;font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;color: rgb(68, 68, 68);font-size: 18px;line-height: 24px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);"><strong>三、利用消息唯一ID</strong></h3> <p style="margin-top: 15px;margin-bottom: 15px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);">每条RabbitMQ消息都有一个唯一的ID（messageId），可以利用这个ID来避免消息的重复消费。具体方法包括：</p> <ol style="margin-top: 15px;margin-bottom: 15px;padding-left: 30px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);" class="list-paddingleft-1"> <li><p><span style="font-weight: 700;">记录已处理消息的ID</span>：消费者可以在处理消息前，先检查该消息的ID是否已经被处理过。这可以通过将已处理消息的ID存储在数据库、Redis等持久化存储中来实现。如果消息的ID已存在，则忽略该消息；否则，进行正常处理并记录该ID。</p></li> <li><p><span style="font-weight: 700;">使用消息去重表</span>：在数据库中维护一个消息去重表，存储每条消息的唯一ID及其处理状态。消费者每次接收到消息时，先查询该ID是否已存在于去重表中，如果存在且状态为“已处理”，则直接丢弃该消息；否则，进行正常处理并更新去重表。</p></li> </ol> <h3 style="margin-top: 25px;margin-bottom: 15px;font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;color: rgb(68, 68, 68);font-size: 18px;line-height: 24px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);"><strong>四、使用第三方插件</strong></h3> <p style="margin-top: 15px;margin-bottom: 15px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);">RabbitMQ社区提供了一些第三方的消息去重插件，如rabbitmq-message-deduplication、rabbitmq-deduplication等。这些插件提供了更专业的消息去重功能，可以根据具体的业务需求进行配置和使用。</p> <h3 style="margin-top: 25px;margin-bottom: 15px;font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;color: rgb(68, 68, 68);font-size: 18px;line-height: 24px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);"><strong>五、合理设置消息的重试机制和过期时间</strong></h3> <ol style="margin-top: 15px;margin-bottom: 15px;padding-left: 30px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);" class="list-paddingleft-1"> <li><p><span style="font-weight: 700;">重试机制</span>：RabbitMQ支持消息的重试机制。通过设置合适的重试次数，可以减少因消费者异常导致的重复消费风险。当消息处理失败时，RabbitMQ会将消息重新投递给消费者进行重试。如果重试次数达到上限，消息将被丢弃或投递到死信队列进行后续处理。</p></li> <li><p><span style="font-weight: 700;">过期时间</span>：RabbitMQ支持为消息设置过期时间。当消息在队列中存留的时间超过设定的过期时间时，消息将被自动删除。这有助于避免因消息长时间未处理而导致的重复消费问题。</p></li> </ol> <h3 style="margin-top: 25px;margin-bottom: 15px;font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;color: rgb(68, 68, 68);font-size: 18px;line-height: 24px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);"><strong>六、其他注意事项</strong></h3> <ol style="margin-top: 15px;margin-bottom: 15px;padding-left: 30px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);" class="list-paddingleft-1"> <li><p><span style="font-weight: 700;">消费者故障处理</span>：在多个消费者共享同一队列时，如果某个消费者未能正确发送确认消息或发生故障，RabbitMQ可能会将消息重新分配给其他消费者。因此，需要确保消费者在处理消息时具有足够的健壮性和容错性。</p></li> <li><p><span style="font-weight: 700;">持久化配置</span>：为了确保消息在RabbitMQ服务重启后不会丢失，可以将队列和消息设置为持久化。这样即使RabbitMQ服务发生故障并重启，已经发送但尚未处理的消息仍然可以被消费者继续处理。</p></li> </ol> <p style="margin-top: 15px;margin-bottom: 15px;color: rgb(68, 68, 68);font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif;font-size: 14px;letter-spacing: normal;text-align: start;background-color: rgb(255, 255, 255);">综上所述，RabbitMQ通过消息确认机制、幂等性设计、利用消息唯一ID、使用第三方插件、合理设置消息的重试机制和过期时间等多种策略来尽量避免消息的重复消费。在设计系统时，应根据具体的业务需求和系统架构选择合适的策略和方法。</p>

<p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">虚拟线程是 Java 21 引入的一个新特性，用于简化并发编程。它与传统的操作系统线程相比，具有显著的优势：</p> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">轻量级</span>：虚拟线程由 JVM 管理，而非操作系统，因此它们的内存占用和创建成本远低于传统线程。理论上，你可以轻松创建数十万甚至更多的虚拟线程。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">高并发性</span>：虚拟线程能处理更高并发的场景，特别是 I/O 密集型的应用，适合开发高并发、响应式的应用程序。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">自动管理</span>：无需手动管理线程池，JVM 会根据负载自动调整虚拟线程的调度，简化了并发编程的复杂性。 </section></li> </ol> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">虚拟线程的基础用法</span></h2> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">创建虚拟线程非常简单。你可以像创建传统线程一样启动虚拟线程，但它的创建与启动更加轻量：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer">Thread virtualThread = Thread.ofVirtual().start(() -&gt; {</span></code><code><span class="code-snippet_outer"> System.<span class="code-snippet__keyword">out</span>.println(<span class="code-snippet__string">"虚拟线程正在运行"</span>);</span></code><code><span class="code-snippet_outer">});</span></code><code><span class="code-snippet_outer">System.<span class="code-snippet__keyword">out</span>.println(<span class="code-snippet__string">"主线程正在运行"</span>);</span></code></pre> </section> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">虚拟线程的延迟启动：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer">Thread virtualThread = Thread.ofVirtual()</span></code><code><span class="code-snippet_outer"> .name(<span class="code-snippet__string">"虚拟线程"</span>)</span></code><code><span class="code-snippet_outer"> .unstarted(() -&gt; System.<span class="code-snippet__keyword">out</span>.println(<span class="code-snippet__string">"虚拟线程运行中"</span>));</span></code><code><span class="code-snippet_outer"><br></span></code><code><span class="code-snippet_outer">virtualThread.start();</span></code><code><span class="code-snippet_outer">virtualThread.<span class="code-snippet__keyword">join</span>(); <span class="code-snippet__comment">// 等待虚拟线程完成</span></span></code></pre> </section> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">在Spring Boot中使用虚拟线程</span></h2> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">在 Spring Boot 项目中使用虚拟线程需要一些简单的配置：</p> <section style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;word-spacing: 0.8px;background-color: rgb(255, 255, 255);text-align: center;margin-bottom: 16px;"> <img class="rich_pages wxw-img" src="https://studyjava.cn/files/articleCover/zsljava11733930693738884.png" data-cropx1="0" data-cropx2="941.6760563380282" data-cropy1="0" data-cropy2="445.43309859154925" data-galleryid="" data-imgfileid="100045784" data-ratio="0.4729011689691817" data-s="300,640" src="/upload/f04f513fabb195d2446283737613f997.png" data-type="jpeg" data-w="941" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 556px !important;visibility: visible !important;"> </section> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;margin-bottom: 16px;"> <span style="font-size: 16px;"><span style="font-size: 16px;-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">确保 Java 版本为 21 或以上</span>。</span> </section> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 在&nbsp; <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">pom.xml</code>&nbsp;中启用&nbsp; <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">--enable-preview</code>，以便支持虚拟线程特性。 </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="xml"><code><span class="code-snippet_outer"><span class="code-snippet__tag">&lt;<span class="code-snippet__name">plugin</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;<span class="code-snippet__name">groupId</span>&gt;</span>org.apache.maven.plugins<span class="code-snippet__tag">&lt;/<span class="code-snippet__name">groupId</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;<span class="code-snippet__name">artifactId</span>&gt;</span>maven-compiler-plugin<span class="code-snippet__tag">&lt;/<span class="code-snippet__name">artifactId</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;<span class="code-snippet__name">configuration</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;<span class="code-snippet__name">source</span>&gt;</span>21<span class="code-snippet__tag">&lt;/<span class="code-snippet__name">source</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;<span class="code-snippet__name">target</span>&gt;</span>21<span class="code-snippet__tag">&lt;/<span class="code-snippet__name">target</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;<span class="code-snippet__name">compilerArgs</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;<span class="code-snippet__name">arg</span>&gt;</span>--enable-preview<span class="code-snippet__tag">&lt;/<span class="code-snippet__name">arg</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;/<span class="code-snippet__name">compilerArgs</span>&gt;</span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag">&lt;/<span class="code-snippet__name">configuration</span>&gt;</span></span></code><code><span class="code-snippet_outer"><span class="code-snippet__tag">&lt;/<span class="code-snippet__name">plugin</span>&gt;</span></span></code></pre> </section> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 在&nbsp; <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">application.properties</code>&nbsp;中启用性能监控工具： </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="makefile"><code><span class="code-snippet_outer">management.endpoints.web.exposure.<span class="code-snippet__keyword">include</span>=health,info,metrics</span></code></pre> </section> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 在 Spring Boot 中为 Tomcat 配置虚拟线程执行器： </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="css"><code><span class="code-snippet_outer">@<span class="code-snippet__keyword">Bean</span></span></code><code><span class="code-snippet_outer">public TomcatProtocolHandlerCustomizer&lt;?&gt; protocolHandlerVirtualThreadExecutorCustomizer() {</span></code><code><span class="code-snippet_outer"> <span class="code-snippet__selector-tag">return</span> <span class="code-snippet__selector-tag">protocolHandler</span> <span class="code-snippet__selector-tag">-</span>&gt; <span class="code-snippet__selector-tag">protocolHandler</span><span class="code-snippet__selector-class">.setExecutor</span>(<span class="code-snippet__selector-tag">Executors</span><span class="code-snippet__selector-class">.newVirtualThreadPerTaskExecutor</span>());</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">实验：传统线程 vs 虚拟线程</span></h2> <h3 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 30px;margin-bottom: 15px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(34, 34, 34);line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1. 创建100,000个线程并执行</span></h3> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">传统线程</span>：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer"><span class="code-snippet__keyword">for</span> (<span class="code-snippet__keyword">int</span> i = <span class="code-snippet__number">0</span>; i &lt; <span class="code-snippet__number">100</span>_000; i++) {</span></code><code><span class="code-snippet_outer"> Thread thread = <span class="code-snippet__keyword">new</span> Thread(() -&gt; System.<span class="code-snippet__keyword">out</span>.println(i));</span></code><code><span class="code-snippet_outer"> thread.start();</span></code><code><span class="code-snippet_outer"> thread.<span class="code-snippet__keyword">join</span>();</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">执行耗时约&nbsp;<span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">18.6 秒</span>。</p> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">虚拟线程</span>：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer"><span class="code-snippet__keyword">for</span> (<span class="code-snippet__keyword">int</span> i = <span class="code-snippet__number">0</span>; i &lt; <span class="code-snippet__number">100</span>_000; i++) {</span></code><code><span class="code-snippet_outer"> Thread thread = Thread.ofVirtual().unstarted(() -&gt; System.<span class="code-snippet__keyword">out</span>.println(i));</span></code><code><span class="code-snippet_outer"> thread.start();</span></code><code><span class="code-snippet_outer"> thread.<span class="code-snippet__keyword">join</span>();</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">执行耗时仅&nbsp;<span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">3.7 秒</span>，性能提升了近&nbsp;<span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">500%</span>&nbsp;。</p> <h3 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 30px;margin-bottom: 15px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(34, 34, 34);line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2. HTTP 请求性能对比</span></h3> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">在高并发场景下，虚拟线程的优势尤为明显。我们对比了传统线程与虚拟线程在处理 HTTP 请求时的表现。</p> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">配置 HTTP 线程执行器</span>：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="css"><code><span class="code-snippet_outer">@<span class="code-snippet__keyword">Bean</span></span></code><code><span class="code-snippet_outer">public TomcatProtocolHandlerCustomizer&lt;?&gt; protocolHandlerVirtualThreadExecutorCustomizer() {</span></code><code><span class="code-snippet_outer"> <span class="code-snippet__selector-tag">return</span> <span class="code-snippet__selector-tag">protocolHandler</span> <span class="code-snippet__selector-tag">-</span>&gt; <span class="code-snippet__selector-tag">protocolHandler</span><span class="code-snippet__selector-class">.setExecutor</span>(<span class="code-snippet__selector-tag">Executors</span><span class="code-snippet__selector-class">.newVirtualThreadPerTaskExecutor</span>());</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">请求测试</span>：发送 1600 个 HTTP 请求，400 并发。</p> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">传统线程</span>： </section></li> <ul class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;list-style-type: disc;"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 请求耗时： <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">9.659 秒</span> </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 每秒请求数： <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">165.65</span> </section></li> </ul> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">虚拟线程</span>： </section></li> <ul class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;list-style-type: disc;"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 请求耗时： <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">7.912 秒</span> </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 每秒请求数： <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">202.22</span> </section></li> </ul> </ol> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">虚拟线程的吞吐量大幅提升，响应时间显著缩短。</p> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">Java性能提升的其他技巧</span></h2> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">除了虚拟线程，Java 还有一些其他的性能提升技巧，尤其适用于 Spring Boot 高并发场景：</p> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">使用并行流</span>：对于 CPU 密集型任务，可以使用并行流（ <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">parallelStream()</code>）来利用多核 CPU，提高处理速度。 </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="typescript"><code><span class="code-snippet_outer">List&lt;Integer&gt; numbers = Arrays.asList(<span class="code-snippet__number">1</span>, <span class="code-snippet__number">2</span>, <span class="code-snippet__number">3</span>, <span class="code-snippet__number">4</span>, <span class="code-snippet__number">5</span>);</span></code><code><span class="code-snippet_outer">numbers.parallelStream().forEach(<span class="code-snippet__built_in">number</span> -&gt; {</span></code><code><span class="code-snippet_outer"> System.out.println(<span class="code-snippet__built_in">number</span> * <span class="code-snippet__number">2</span>);</span></code><code><span class="code-snippet_outer">});</span></code></pre> </section> <ol start="2" data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">异步编程与CompletableFuture</span>：对于 I/O 密集型任务，可以使用&nbsp; <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">CompletableFuture</code>&nbsp;进行异步处理，减少线程阻塞，提高响应性能。 </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer">CompletableFuture&lt;Void&gt; future = CompletableFuture.runAsync(() -&gt; {</span></code><code><span class="code-snippet_outer"> <span class="code-snippet__comment">// 异步执行任务</span></span></code><code><span class="code-snippet_outer"> System.<span class="code-snippet__keyword">out</span>.println(<span class="code-snippet__string">"异步任务完成"</span>);</span></code><code><span class="code-snippet_outer">});</span></code><code><span class="code-snippet_outer">future.<span class="code-snippet__keyword">join</span>(); <span class="code-snippet__comment">// 等待任务完成</span></span></code></pre> </section> <ol start="3" data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">优化数据库查询</span>：减少数据库查询的次数，使用缓存（如 Redis）来存储频繁查询的数据，减少不必要的 I/O 操作。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">内存管理优化</span>：通过使用对象池（如&nbsp; <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">Apache Commons Pool</code>）来管理资源，减少频繁的对象创建和销毁，提高内存使用效率。 </section></li> </ol> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">小结</span></h2> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">虚拟线程</span>&nbsp;是 Java 并发编程的革新，它简化了线程管理，提升了高并发场景下的性能。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 使用虚拟线程，你可以轻松创建数十万甚至更多线程，而不会影响应用的性能。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 在 Spring Boot 中配置虚拟线程非常简单，只需几行代码即可启用虚拟线程，带来显著的性能提升。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 除了虚拟线程，其他优化技巧（如并行流、异步编程、数据库查询优化等）也能有效提升 Java 应用的性能。 </section></li> </ol> <section style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, &quot;PingFang SC&quot;, Cambria, Cochin, Georgia, Times, &quot;Times New Roman&quot;, serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;margin-bottom: 24px;"> 通过这些技巧，Spring Boot 应用能够在高并发场景下表现出更强的性能和更低的响应延迟。 </section>

<section data-tool="mdnice编辑器" data-website="https://www.mdnice.com" style="margin-bottom: 0px;padding-left: 10px;padding-right: 10px;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;font-family: Optima, &quot;Microsoft YaHei&quot;, PingFangSC-regular, serif;font-size: 16px;color: rgb(0, 0, 0);line-height: 1.5em;word-spacing: 0em;letter-spacing: 0em;word-break: break-word;text-align: left;"> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">是一个很受欢迎的内存消息队列，它源于 <span style="letter-spacing: normal;">LMAX </span>对并发、性能和非阻塞算法的研究。今天一起来学习一下这个消息队列。</span></p> <h2 data-tool="mdnice编辑器" style="border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(239, 112, 96);margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none none solid;border-width: 1px 1px 2px;border-radius: 0px;box-shadow: none;display: flex;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.1em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="font-size: 22px;color: rgb(255, 255, 255);background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(239, 112, 96);line-height: 1.5em;letter-spacing: 0em;align-items: unset;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 3px 3px 0px 0px;box-shadow: none;display: inline-block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;margin-right: 5px;overflow: unset;padding: 3px 10px 1px;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">简介</span></h2> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">对于主流的分布式消息队列来说，一般会包含<span style="letter-spacing: normal;"> Producer、Broker、Consumer</span>、注册中心等模块。比如 <span style="letter-spacing: normal;">RocketMQ </span>架构如下：</span></p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100006568" data-ratio="0.4856278366111952" src="/upload/a92fda774fcaacaf60cc861cb5f19d27.png" data-type="png" data-w="661" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">并不是分布式消息队列，它是一款内存消息队列，因此架构上跟分布式消息队列有很大差别。下面是一张 <span style="letter-spacing: normal;">LMAX </span>使用 <span style="letter-spacing: normal;">Disruptor </span>的案例图：</span></p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100006569" data-ratio="0.6650124069478908" src="/upload/46f24888fd487963438dfbc5b4e3b4d7.png" data-type="png" data-w="806" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">我们介绍一下 <span style="letter-spacing: normal;">Disruptor </span>架构中的核心概念。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.1 Ring Buffer</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">Ring Buffer 通常被认为是 <span style="letter-spacing: normal;">Disruptor </span>的最主要的设计，但是从 3.0 版本开始，<span style="letter-spacing: normal;">Ring Buffer</span> 只负责存储和更新经过 <span style="letter-spacing: normal;">Disruptor </span>的数据。在一些高级的使用场景，它甚至完全可以被用户替换。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.2 Sequence</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">使用 <span style="letter-spacing: normal;">Sequence </span>来识别特定组件的位置。每个 <span style="letter-spacing: normal;">Consumer</span>（也就是事件处理器）都像 <span style="letter-spacing: normal;">Disruptor </span>一样持有一个 <span style="letter-spacing: normal;">Sequence</span>。并发相关的核心代码依赖 <span style="letter-spacing: normal;">Sequence </span>的自增值，因此 <span style="letter-spacing: normal;">Sequence </span>具有跟 <span style="letter-spacing: normal;">AtomicLong </span>相似的特性，事实上唯一的不同就是不同的 <span style="letter-spacing: normal;">Sequence </span>之间不存在伪共享问题。</span></p> <blockquote data-tool="mdnice编辑器" style="margin-top: 20px;margin-bottom: 20px;padding: 10px 10px 10px 20px;border-top: 3px none rgba(0, 0, 0, 0.4);border-bottom: 3px none rgba(0, 0, 0, 0.4);border-right: 3px none rgba(0, 0, 0, 0.4);border-left-color: rgb(239, 112, 96);border-radius: 0px;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(255, 249, 249);width: auto;height: auto;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;overflow: auto;"> <p style="text-indent: 0em;padding-top: 8px;padding-bottom: 8px;color: rgb(0, 0, 0);line-height: 1.8em;letter-spacing: 0px;"><span style="letter-spacing: 1px;">伪共享：<span style="letter-spacing: normal;">CPU </span>缓存是以缓存行为单位进行加载和存储，<span style="letter-spacing: normal;">CPU </span>每次从主存中拉取数据时，会把相邻的数据也存入同一个缓存行。即使多个线程操作的是同一缓存行中不同的变量，只要有一个线程修改了缓存行中的某一个变量值，该缓存行就会被标记为无效，需要重新从主从中加载。在多线程环境下，频繁地重新加载缓存行，会严重影响了程序执行效率。</span></p> </blockquote> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.3 Sequencer</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Sequencer </span><span style="letter-spacing: 1px;">是 <span style="letter-spacing: normal;">Disrupter </span>的真正核心，有单个生产者和多个生产者两种实现（<span style="letter-spacing: normal;">SingleProducerSequencer </span>和 <span style="letter-spacing: normal;">MultiProducerSequencer</span>）。为了让数据在生产者和消费者之间快速、准确地传输，它们都实现了所有并发算法。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.4 Sequence Barrier</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Sequencer </span><span style="letter-spacing: 1px;">生成一个<span style="letter-spacing: normal;"> Sequence Barrier</span>，它包含由 <span style="letter-spacing: normal;">Sequencer&nbsp;</span>生成的 <span style="letter-spacing: normal;">Sequence </span>和消费者拥有的 <span style="letter-spacing: normal;">Sequence </span>的引用。<span style="letter-spacing: normal;">Sequence Barrier </span>决定是否有事件给消费者处理。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.5 Wait Strategy</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">消费者怎样等待事件的到来。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.6 Event Processor</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">主要负责循环处理来自 <span style="letter-spacing: normal;">Disruptor </span>事件，它拥有消费者 <span style="letter-spacing: normal;">Sequence </span>的所有权。有一个单独的实现类 <span style="letter-spacing: normal;">BatchEventProcessor</span>，这个类拥有高效的事件循环处理能力并且处理完成后可以回调实现 <span style="letter-spacing: normal;">EventHandler </span>接口的用户。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.7 Event Handler</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">由用户来实现并且代表 <span style="letter-spacing: normal;">Disruptor </span>消费者的接口。</span></p> <h2 data-tool="mdnice编辑器" style="border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(239, 112, 96);margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none none solid;border-width: 1px 1px 2px;border-radius: 0px;box-shadow: none;display: flex;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.1em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="font-size: 22px;color: rgb(255, 255, 255);background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(239, 112, 96);line-height: 1.5em;letter-spacing: 0em;align-items: unset;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 3px 3px 0px 0px;box-shadow: none;display: inline-block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;margin-right: 5px;overflow: unset;padding: 3px 10px 1px;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">2 Disruptor 特性</span></h2> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2.1 多播事件</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">多播事件是 <span style="letter-spacing: normal;">Disruptor </span>区别于其他队列的最大差异。其他队列都是一个事件消息只能被单个消费者消费，而 <span style="letter-spacing: normal;">Disruptor </span>如果有多个消费者监听，则可以将所有事件消息发送给所有消费者。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">在前面 <span style="letter-spacing: normal;">LMAX </span>使用 <span style="letter-spacing: normal;">Disruptor </span>的案例图中，有 <span style="letter-spacing: normal;">JournalConsumer</span>、<span style="letter-spacing: normal;">ReplicationConsumer </span>和 <span style="letter-spacing: normal;">ApplicationConsumer </span>三个消费者监听了 <span style="letter-spacing: normal;">Disruptor</span>，这三个消费者将收到来了 <span style="letter-spacing: normal;">Disruptor </span>的所有消息。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2.2 消费者依赖关系图</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">为了支持并发处理在实际业务场景中的需要，有时消费者直接需要做协调。再回到前面 <span style="letter-spacing: normal;">LMAX </span>使用 <span style="letter-spacing: normal;">Disruptor </span>的案例，在 <span style="letter-spacing: normal;">journalling </span>和 <span style="letter-spacing: normal;">replication </span>这两个消费者处理完成之前，有必要阻止业务逻辑消费者开始处理。我们称这个特征为“<span style="letter-spacing: normal;">gating</span>”（或者更准确地说，该特征是“gating”的一种形式）。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">首先，确保生产者数量不会超过消费者。这通过调用 <span style="letter-spacing: normal;">RingBuffer.addGatingConsumers（）</span>来将相关消费者添加到 <span style="letter-spacing: normal;">Disruptor</span>。其次，消费者依赖关系的实现是通过构建一个 <span style="letter-spacing: normal;">SequenceBarrier</span>，<span style="letter-spacing: normal;">SequenceBarrier </span>拥有需要在它前面完成处理逻辑的消费者的 <span style="letter-spacing: normal;">Sequence</span>。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">就拿前面&nbsp;L<span style="letter-spacing: normal;">MAX </span>使用 <span style="letter-spacing: normal;">Disruptor </span>的案例来说，<span style="letter-spacing: normal;">ApplicationConsumer </span>的 <span style="letter-spacing: normal;">SequenceBarrier </span>拥有 <span style="letter-spacing: normal;">JournalConsumer </span>和 <span style="letter-spacing: normal;">ReplicationConsumer </span>这 2 个消费者的 <span style="letter-spacing: normal;">Sequence</span>，所以 <span style="letter-spacing: normal;">ApplicationConsumer </span>对 <span style="letter-spacing: normal;">JournalConsumer </span>和 <span style="letter-spacing: normal;">ReplicationConsumer </span>的依赖关系可以从 <span style="letter-spacing: normal;">SequenceBarrier </span>到 <span style="letter-spacing: normal;">Sequence </span>的连接中看到。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Sequencer </span><span style="letter-spacing: 1px;">和下游消费者的关系也需要注意。<span style="letter-spacing: normal;">Sequencer </span>的一个角色就是发布的事件消息不能超出<span style="letter-spacing: normal;"> Ring Buffer</span>。这就要求下游消费者的 <span style="letter-spacing: normal;">Sequence </span>不能小于<span style="letter-spacing: normal;"> Ring Buffer </span>的<span style="letter-spacing: normal;"> Sequence</span>，也不能小于<span style="letter-spacing: normal;"> Ring Buffer </span>的大小。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">上面图中，因为<span style="letter-spacing: normal;"> ApplicationConsumer </span>的<span style="letter-spacing: normal;"> Sequence </span>必须要保证小于等于 <span style="letter-spacing: normal;">JournalConsumer </span>和<span style="letter-spacing: normal;"> ReplicationConsumer </span>的<span style="letter-spacing: normal;"> Sequence</span>，因此 <span style="letter-spacing: normal;">Sequencer </span>只需要关心<span style="letter-spacing: normal;"> ApplicationConsumer </span>的<span style="letter-spacing: normal;"> Sequence</span>。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2.3 内存预分配</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">的目标是低延迟，因此减少或者去除内存分配是必要的。在基于 <span style="letter-spacing: normal;">Java </span>的系统中，目标是减少 <span style="letter-spacing: normal;">STW </span>次数。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">为了支持这一点，用户可以在 <span style="letter-spacing: normal;">Disruptor </span>中预分配事件所需的内存。在预分配内存时，用户提供的<span style="letter-spacing: normal;"> EventFactory </span>将对<span style="letter-spacing: normal;"> Ring Buffer </span>的所有元素进行调用。当生产者向<span style="letter-spacing: normal;"> Disruptor </span>发送新的事件消息时，<span style="letter-spacing: normal;">Disruptor </span>的<span style="letter-spacing: normal;"> API </span>允许用户使用构造好的对象，他们可以调用对象的方法或者更新对象的字段。<span style="letter-spacing: normal;">Disruptor </span>需要确保并发安全。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2.4 无锁并发</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">实现低延迟的另一个关键方法时使用无锁算法，通过使用内存屏障和<span style="letter-spacing: normal;"> CAS </span>来实现内存可见性和正确性。<span style="letter-spacing: normal;">Disruptor </span>唯一使用锁的地方就是在<span style="letter-spacing: normal;"> BlockingWaitStrategy</span>。</span></p> <h2 data-tool="mdnice编辑器" style="border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(239, 112, 96);margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none none solid;border-width: 1px 1px 2px;border-radius: 0px;box-shadow: none;display: flex;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.1em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="font-size: 22px;color: rgb(255, 255, 255);background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(239, 112, 96);line-height: 1.5em;letter-spacing: 0em;align-items: unset;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 3px 3px 0px 0px;box-shadow: none;display: inline-block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;margin-right: 5px;overflow: unset;padding: 3px 10px 1px;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">3 调优选项</span></h2> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">虽然大多数场景下<span style="letter-spacing: normal;"> Disruptor </span>可以表现出优秀的性能，但是仍然有一些调优参数可以改进<span style="letter-spacing: normal;"> Disruptor </span>的性能。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">3.1 单个/多个生产者</span></h3> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;">Disruptor&lt;LongEvent&gt;&nbsp;disruptor&nbsp;=&nbsp;<span style="color: #c678dd;line-height: 26px;">new</span>&nbsp;Disruptor(<br>&nbsp;factory,<br>&nbsp;bufferSize,<br>&nbsp;DaemonThreadFactory.INSTANCE,<br>&nbsp;ProducerType.SINGLE,&nbsp;<br>&nbsp;<span style="color: #c678dd;line-height: 26px;">new</span>&nbsp;BlockingWaitStrategy()&nbsp;<br>);<br></code></pre> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">上面是<span style="letter-spacing: normal;"> disruptor </span>的构造函数，<span style="letter-spacing: normal;">ProducerType.SINGLE </span>表示创建单生产者的 <span style="letter-spacing: normal;">Sequencer</span>，<span style="letter-spacing: normal;">ProducerType.MULTI &nbsp;</span>表示创建多生产者的<span style="letter-spacing: normal;"> Sequencer</span>。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">在并发系统中提高系统性能的最好方式是遵循单写原则。下面是官方的一个 <span style="letter-spacing: normal;">disruptor </span>吞吐量测试结果，测试环境是<span style="letter-spacing: normal;"> i7 Sandy Bridge MacBook Air。</span></span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">单生产者：</span></p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100006570" data-ratio="1.0259067357512954" src="/upload/ec8a0f8df318ff27fa65974153968463.jpg" data-type="jpeg" data-w="386" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">多生产者：</span></p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100006571" data-ratio="1.0335917312661498" src="/upload/2116b2504923610e283a8576edd9b022.jpg" data-type="jpeg" data-w="387" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">3.2 等待策略</span></h3> <ol data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: normal;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: normal;">BlockingWaitStrategy</span> <span style="letter-spacing: 1px;"></span> </section></li> </ol> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">disruptor </span><span style="letter-spacing: 1px;">的默认等待策略是 <span style="letter-spacing: normal;">BlockingWaitStrategy</span>，这种策略使用锁和唤醒锁的 <span style="letter-spacing: normal;">Condition </span>变量。</span></p> <ol start="2" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: normal;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: normal;">SleepingWaitStrategy</span> <span style="letter-spacing: 1px;"></span> </section></li> </ol> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">跟 <span style="letter-spacing: normal;">BlockingWaitStrategy </span>策略类似，他是通过 <span style="letter-spacing: normal;">LockSupport.parkNanos(1) </span>方法来实现等待，不需要给 <span style="letter-spacing: normal;">Condition </span>变量发送信号来唤醒等待。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">主要适用于对延时要求不高的场景，比如异步打印日志。</span></p> <ol start="3" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: normal;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: normal;">YieldingWaitStrategy</span> <span style="letter-spacing: 1px;"></span> </section></li> </ol> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">‌<span style="letter-spacing: normal;">YieldingWaitStrategy </span>策略使用<span style="letter-spacing: normal;"> Busy spin‌</span>（不释放&nbsp;<span style="letter-spacing: normal;">CPU&nbsp;</span>资源，通过循环检查条件直到条件满足为止）技术来等待 <span style="letter-spacing: normal;">sequence </span>增长到一个合适的值。在循环内部会调用 Thread#yield() 方法允许其他排队线程去执行。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">这种策略主要用于通过消耗 <span style="letter-spacing: normal;">CPU </span>来实现低延迟的场景。当 <span style="letter-spacing: normal;">EventHandler </span>数量消息逻辑 <span style="letter-spacing: normal;">CPU </span>核数并且对延迟要求较高时，可以考虑这种等待策略。</span></p> <ol start="4" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: normal;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: normal;">BusySpinWaitStrategy</span> <span style="letter-spacing: 1px;"></span> </section></li> </ol> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">BusySpinWaitStrategy </span><span style="letter-spacing: 1px;">是性能最高的等待策略，它适用于低延迟系统，但是对部署环境要求很高。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">这种等待策略的唯一适用场景是当 <span style="letter-spacing: normal;">EventHandler </span>数量消息逻辑 <span style="letter-spacing: normal;">CPU </span>核数并且超线程被禁用。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">4 官方示例</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">下面是一个官方示例。这个例子比较简单，就是生产者向消费者发送一个 <span style="letter-spacing: normal;">long </span>类型的值。</span></p> <ol data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: 1px;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: 1px;">首先定义一个 <span style="letter-spacing: normal;">Event</span>。</span> </section></li> </ol> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span>&nbsp;<span style="color: #e6c07b;line-height: 26px;">LongEvent</span><br></span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">private</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">long</span>&nbsp;value;<br><br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">void</span>&nbsp;<span style="color: #61aeee;line-height: 26px;">set</span><span style="line-height: 26px;">(<span style="color: #c678dd;line-height: 26px;">long</span>&nbsp;value)</span><br>&nbsp;&nbsp;&nbsp;&nbsp;</span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">this</span>.value&nbsp;=&nbsp;value;<br>&nbsp;&nbsp;&nbsp;&nbsp;}<br><br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #61aeee;line-height: 26px;">@Override</span><br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;String&nbsp;<span style="color: #61aeee;line-height: 26px;">toString</span><span style="line-height: 26px;">()</span><br>&nbsp;&nbsp;&nbsp;&nbsp;</span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">return</span>&nbsp;<span style="color: #98c379;line-height: 26px;">"LongEvent{"</span>&nbsp;+&nbsp;<span style="color: #98c379;line-height: 26px;">"value="</span>&nbsp;+&nbsp;value&nbsp;+&nbsp;<span style="color: #98c379;line-height: 26px;">'}'</span>;<br>&nbsp;&nbsp;&nbsp;&nbsp;}<br>}<br></code></pre> <ol start="2" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: 1px;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: 1px;">为了能让 <span style="letter-spacing: normal;">Disruptor </span>预分配内存，这里定义一个 <span style="letter-spacing: normal;">LongEventFactory</span>。</span> </section></li> </ol> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span>&nbsp;<span style="color: #e6c07b;line-height: 26px;">LongEventFactory</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">implements</span>&nbsp;<span style="color: #e6c07b;line-height: 26px;">EventFactory</span>&lt;<span style="color: #e6c07b;line-height: 26px;">LongEvent</span>&gt;<br></span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #61aeee;line-height: 26px;">@Override</span><br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;LongEvent&nbsp;<span style="color: #61aeee;line-height: 26px;">newInstance</span><span style="line-height: 26px;">()</span><br>&nbsp;&nbsp;&nbsp;&nbsp;</span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">return</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">new</span>&nbsp;LongEvent();<br>&nbsp;&nbsp;&nbsp;&nbsp;}<br>}<br></code></pre> <ol start="3" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: 1px;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: 1px;">创建一个消费者来处理事件</span> </section></li> </ol> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span>&nbsp;<span style="color: #e6c07b;line-height: 26px;">LongEventHandler</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">implements</span>&nbsp;<span style="color: #e6c07b;line-height: 26px;">EventHandler</span>&lt;<span style="color: #e6c07b;line-height: 26px;">LongEvent</span>&gt;<br></span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #61aeee;line-height: 26px;">@Override</span><br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">void</span>&nbsp;<span style="color: #61aeee;line-height: 26px;">onEvent</span><span style="line-height: 26px;">(LongEvent&nbsp;event,&nbsp;<span style="color: #c678dd;line-height: 26px;">long</span>&nbsp;sequence,&nbsp;<span style="color: #c678dd;line-height: 26px;">boolean</span>&nbsp;endOfBatch)</span><br>&nbsp;&nbsp;&nbsp;&nbsp;</span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.out.println(<span style="color: #98c379;line-height: 26px;">"Event:&nbsp;"</span>&nbsp;+&nbsp;event);<br>&nbsp;&nbsp;&nbsp;&nbsp;}<br>}<br></code></pre> <ol start="4" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: 1px;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: 1px;">编写发送事件消息的逻辑</span> </section></li> </ol> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">import</span>&nbsp;com.lmax.disruptor.dsl.Disruptor;<br><span style="color: #c678dd;line-height: 26px;">import</span>&nbsp;com.lmax.disruptor.RingBuffer;<br><span style="color: #c678dd;line-height: 26px;">import</span>&nbsp;com.lmax.disruptor.examples.longevent.LongEvent;<br><span style="color: #c678dd;line-height: 26px;">import</span>&nbsp;com.lmax.disruptor.util.DaemonThreadFactory;<br><span style="color: #c678dd;line-height: 26px;">import</span>&nbsp;java.nio.ByteBuffer;<br><br><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span>&nbsp;<span style="color: #e6c07b;line-height: 26px;">LongEventMain</span><br></span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;<span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">static</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">void</span>&nbsp;<span style="color: #61aeee;line-height: 26px;">main</span><span style="line-height: 26px;">(String[]&nbsp;args)</span>&nbsp;<span style="color: #c678dd;line-height: 26px;">throws</span>&nbsp;Exception<br>&nbsp;&nbsp;&nbsp;&nbsp;</span>{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">int</span>&nbsp;bufferSize&nbsp;=&nbsp;<span style="color: #d19a66;line-height: 26px;">1024</span>;&nbsp;<br><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Disruptor&lt;LongEvent&gt;&nbsp;disruptor&nbsp;=&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">new</span>&nbsp;Disruptor&lt;&gt;(LongEvent::<span style="color: #c678dd;line-height: 26px;">new</span>,&nbsp;bufferSize,&nbsp;DaemonThreadFactory.INSTANCE);<br><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;disruptor.handleEventsWith((event,&nbsp;sequence,&nbsp;endOfBatch)&nbsp;-&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.out.println(<span style="color: #98c379;line-height: 26px;">"Event:&nbsp;"</span>&nbsp;+&nbsp;event));&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;disruptor.start();&nbsp;<br><br><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;RingBuffer&lt;LongEvent&gt;&nbsp;ringBuffer&nbsp;=&nbsp;disruptor.getRingBuffer();&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ByteBuffer&nbsp;bb&nbsp;=&nbsp;ByteBuffer.allocate(<span style="color: #d19a66;line-height: 26px;">8</span>);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #c678dd;line-height: 26px;">for</span>&nbsp;(<span style="color: #c678dd;line-height: 26px;">long</span>&nbsp;l&nbsp;=&nbsp;<span style="color: #d19a66;line-height: 26px;">0</span>;&nbsp;<span style="color: #c678dd;line-height: 26px;">true</span>;&nbsp;l++)<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bb.putLong(<span style="color: #d19a66;line-height: 26px;">0</span>,&nbsp;l);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ringBuffer.publishEvent((event,&nbsp;sequence,&nbsp;buffer)&nbsp;-&gt;&nbsp;event.set(buffer.getLong(<span style="color: #d19a66;line-height: 26px;">0</span>)),&nbsp;bb);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thread.sleep(<span style="color: #d19a66;line-height: 26px;">1000</span>);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<br>&nbsp;&nbsp;&nbsp;&nbsp;}<br>}<br></code></pre> <h2 data-tool="mdnice编辑器" style="border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(239, 112, 96);margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none none solid;border-width: 1px 1px 2px;border-radius: 0px;box-shadow: none;display: flex;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.1em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="font-size: 22px;color: rgb(255, 255, 255);background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(239, 112, 96);line-height: 1.5em;letter-spacing: 0em;align-items: unset;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 3px 3px 0px 0px;box-shadow: none;display: inline-block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;margin-right: 5px;overflow: unset;padding: 3px 10px 1px;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">5 总结</span></h2> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">作为一款高性能的内存队列，有不少优秀的设计思想值得我们学习，比如内存预分配、无锁并发。同时它的使用非常简单，推荐大家使用。</span></p> </section>

<section data-tool="mdnice编辑器" data-website="https://www.mdnice.com"> <h2 data-sourcepos="21:1-21:70" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">来一个大的介绍：ElasticSearch的性能优化的几大方面</span></h2> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">总的来说，ElasticSearch的性能优化，可以从以下方面的考虑：</span> </section> <ul data-sourcepos="25:1-48:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">硬件层面的优化：</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">机器分配，机器配置，机器内存，机器CPU，机器网络，机器磁盘性能</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">系统层面的优化：</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">文件句柄优化、swap关闭</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">ElasticSearch集群层面的优化</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">合理分配节点，</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">合理分配参加竞选Master的节点</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">ElasticSearch 索引层面的优化，</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">副本数量、索引数量、分片数量</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">ElasticSearch查询层面的优化</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">7 职责分离，全面监控</span> </section></li> </ul> <h2 data-sourcepos="49:1-49:28" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">1&nbsp;硬件层面的优化</span></h2> <h3 data-sourcepos="53:1-53:51" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">1.1 为ES进行 硬件层面的 内存优化：</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch是一个内存密集型应用，合理分配内存对于Elasticsearch 提高性能至关重要:</span> </section> <ul data-sourcepos="57:1-64:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">确保系统内存有足够的空间用于文件缓存，而不单单是JVM堆内存。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">建议将一半的内存分配给Elasticsearch的堆内存，另一半留给操作系统和其他进程。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">为了避免垃圾回收（GC）成为瓶颈，建议设置堆内存的最大值为30GB或更小。可以通过修改</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">jvm.options</span></code> <span style="font-size: 15px;">文件来调整堆内存的大小。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">总之，ES是比较依赖内存的，并且对内存的消耗也很大，内存对ES的重要性甚至是高于CPU的，所以即使是数据量不大的业务，为了保证服务的稳定性，在满足业务需求的前提下，我们仍需考虑留有不少于20%的冗余性能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">根据业务量不同，内存的需求也不同，一般生产建议总体 32内存，其中JVM 不要少于16G， 还有 16G给底层的Luence。</span> </section> <h3 data-sourcepos="71:1-71:44" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">为ES进行 系统层面的 磁盘优化</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于ES来说，磁盘可能是最重要的了，因为数据都是存储在磁盘上的，当然这里说的磁盘指的是磁盘的性能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">磁盘性能往往是硬件性能的瓶颈，木桶效应中的最短板。</span> </section> <h4 data-sourcepos="79:1-79:61" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">方式一：纵向扩展： 使用最高性能的 磁盘</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES 磁盘优化的最为简单，最为粗暴的方式是 纵向扩展，就是使用最高性能的 磁盘，比如SSD 固态硬盘。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES应用可能要面临不间断的大量的数据读取和写入，数据规模非常庞大， 尼恩曾经架构过的一个30个节点ES集群，数据量非常庞大，达到1TB 。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一般来说，速度最快的，也是最昂贵的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">不可能都用SSD硬盘， 一般是 SSD和机械硬盘组合使用。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">所以，可以考虑把节点冷热分离：</span> </section> <ul data-sourcepos="93:1-100:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">“热节点”使用SSD做存储，可以大幅提高系统性能；</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">“冷数据“存储在机械硬盘中，降低成本。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">如果使用磁盘阵列，可以使用RAID 0。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>这里，尼恩给大家来点基础知识， 什么是RAID 0？</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">RAID（Redundant Array of Independent Disks，独立磁盘冗余阵列）是一种将多个物理磁盘组合成一个逻辑单元的数据存储虚拟化技术，其目的是提高数据存储的性能、可靠性或两者兼顾。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">RAID 0 是RAID中一种配置方式，具体含义如下：</span> </section> <ol data-sourcepos="107:1-140:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>RAID 0 定义</strong>：</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">RAID 0 又称为条带化（Striping），它将数据分散存储在两个或多个硬盘上，每个硬盘只存储数据的一部分，从而提高数据的读写速度。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>工作原理</strong>：</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">在RAID 0配置中，数据被分成多个块（称为“条带”），并按顺序存储在多个硬盘上。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">例如，如果有一个4KB的数据块，它会被分成4个1KB的块，分别存储在4个硬盘上。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">读取数据时，因为可以同时从多个硬盘读取，所以读取速度会加快。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">写入数据时也是同理，数据会被分成多个块并行写入多个硬盘。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>性能提升</strong>：</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">RAID 0 的主要优势在于提高数据的读写速度。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">由于数据被分散在多个硬盘上，所以理论上，RAID 0 的读写速度可以达到单个硬盘速度的总和。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>没有冗余</strong>：</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">RAID 0 的缺点是没有数据冗余。如果其中一个硬盘发生故障，整个阵列上的数据都会丢失，因为每个硬盘上只存储数据的一部分，没有副本。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>适用场景</strong>：</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">RAID 0 适用于对性能要求高而对数据安全性要求不高的场景，如视频编辑、大型数据库的临时存储等。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>RAID 0 风险</strong>：</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">由于没有数据冗余，RAID 0 的风险较高。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">一旦硬盘发生故障，整个RAID 0 阵列上的数据都会丢失，因此在使用RAID 0 时需要非常小心，并且定期备份数据。</span> </section></li> </ol> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">总结来说，RAID 0 是一种通过将数据分散存储在多个硬盘上来提高性能的配置方式，但它不提供任何数据冗余，因此使用时需要考虑到数据丢失的风险。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>这里，尼恩给大家再来点基础知识， 看看 RAID 0 和 RAID 1 的区别?</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">RAID 0 和 RAID 1 是两种不同的RAID配置，RAID 0 和 RAID 1 在数据存储、性能和可靠性方面有显著的区别：</span> </section> <ol data-sourcepos="147:1-166:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>RAID 0（条带化）</strong>： 多块磁盘并行读写</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>数据存储</strong>：数据被分割成多个块（条带），然后分散存储在两个或多个硬盘上。这意味着每个硬盘上存储的是数据的不同部分，而不是数据的副本。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>性能</strong>：由于数据的读写操作可以并行进行，RAID 0 可以提供很高的数据传输速率，特别是在写入大量数据时。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>可靠性</strong>：RAID 0 没有数据冗余，任何一个硬盘的故障都会导致整个阵列上的数据丢失。因此，RAID 0 的可靠性最低。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>适用场景</strong>：适合对读写性能要求高、对数据安全性要求不高的应用，如视频处理、大型数据库的临时存储等。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>RAID 1（镜像）</strong>： 多块磁盘冗写，并行读</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>数据存储</strong>：数据在两个硬盘上存储完全相同的副本，即镜像。这意味着数据在两个硬盘上都有备份。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>性能</strong>：RAID 1 的读取性能通常比写入性能要好，因为多个硬盘可以同时读取数据。但是，写入性能可能受到限制，因为数据需要同时写入两个硬盘。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>可靠性</strong>：RAID 1 提供了很好的数据冗余，即使一个硬盘发生故障，另一个硬盘上的数据副本仍然可以保证数据的完整性和可用性。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>适用场景</strong>：适合对数据安全性要求高的应用，如关键业务数据存储、服务器操作系统等。</span> </section></li> </ol> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">总结来说，RAID 0 提供了高性能但牺牲了数据安全性，而RAID 1 则提供了数据冗余和较高的可靠性，但性能相对较低。在选择RAID配置时，需要根据具体的应用需求和预算来决定使用哪种RAID级别。</span> </section> <h4 data-sourcepos="171:1-171:72" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">方式二：横向扩展：使用多块硬盘提高 硬盘的性能</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><br></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">横向 扩展，就是 搞多块磁盘。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch 通过多个</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">path.data</span></code> <span style="font-size: 15px;">目录配置，把Elasticsearch数据条带化分配到使用多块硬盘上面，以提高I/O性能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">要在Elasticsearch中使用多块硬盘，并通过配置多个</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">path.data</span></code> <span style="font-size: 15px;">目录来实现数据的条带化分配，可以按照以下步骤操作：</span> </section> <ol data-sourcepos="185:1-186:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>配置多个数据路径</strong>：</span> </section></li> </ol> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在Elasticsearch的配置文件</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">elasticsearch.yml</span></code> <span style="font-size: 15px;">中，可以指定多个数据存储路径（对应到多个磁盘），用逗号分隔。 例如，如果你有三块硬盘，并且已经分别挂载在</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">/mnt/disk1</span></code> <span style="font-size: 15px;">、</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">/mnt/disk2</span></code> <span style="font-size: 15px;">和</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">/mnt/disk3</span></code> <span style="font-size: 15px;">，你可以这样配置：</span> </section> <pre data-sourcepos="191:4-193:6" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;"><span style="color: rgb(1, 86, 146);">path.data:</span> <span style="color: rgb(84, 121, 13);">/mnt/disk1,/mnt/disk2,/mnt/disk3</span><br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这样配置后，Elasticsearch会尝试在这些路径上存储数据，类似于软件层面的RAID 0配置。</span> </section> <ol start="2" data-sourcepos="197:1-198:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>ES的分片分配机制</strong>：</span> </section></li> </ol> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch在选择存储分片的路径时，会根据一定的逻辑来选择最佳路径。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch会优先选择磁盘使用率较低（/最为空闲）的磁盘来存储新的分片数据文件。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果多个路径可用，Elasticsearch会根据剩余空间和已存储的分片数量来决定将分片分配到哪个路径上。</span> </section> <ol start="3" data-sourcepos="205:1-206:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>数据安全性问题</strong>：</span> </section></li> </ol> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">虽然使用多个</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">path.data</span></code> <span style="font-size: 15px;">路径可以提高I/O性能，但这种做法类似于软件层面的RAID 0，没有在磁盘层面 做 数据冗余。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果其中一个硬盘发生故障，存储在该硬盘上的所有分片都会丢失。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">因此，官方建议，确保每个分片至少有一个副本分片，以防止数据丢失。</span> </section> <h4 data-sourcepos="215:1-215:45" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">系统层面的ES 磁盘优化 总结：</span></h4> <ul data-sourcepos="217:1-225:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">ES应用可能要面临不间断的大量的数据读取和写入，磁盘关系到 ES的性能，建议使用最高速的磁盘设备（如SSD）可以显著提高Elasticsearch的性能。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">当然最高速也是最昂贵的，不可能都用SSD硬盘， 一般是 SSD和机械硬盘组合使用。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">合理规划磁盘布局，可以考虑把节点冷热分离， 将数据按照冷热的程度，存储在不同的磁盘上，可以避免磁盘I/O竞争。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">使用RAID 0可以提高磁盘I/O，但要注意数据的安全性。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">通过多个</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">path.data</span></code> <span style="font-size: 15px;">目录配置，把Elasticsearch数据条带化分配到使用多块硬盘上面，以提高I/O性能。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">避免使用远程挂载的存储，比如NFS或者SMB/CIFS，因为这会引入延迟，影响性能。</span> </section></li> </ul> <h3 data-sourcepos="226:1-226:47" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">1.2 为ES进行 硬件层面的 CPU 调优</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">CPU对其他的 应用程序而言，可谓是最重要的硬件，</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">但对于ES来说，CPU不是他最依赖的硬件， 提升CPU配置可能不会像提升磁盘或者内存配置带来的性能收益更直接、显著。在成本预算一定的前提下，应该把更多的预算花在磁盘以及内存上面。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当然也不是说CPU的性能就不重要，Elasticsearch在处理查询和索引操作时会消耗大量的CPU资源。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">服务器的CPU不需要太高的单核性能，更多的核心数和线程数意味着更高的并发处理能力。现在PC的配置8核都已经普及了，更不用说服务器了。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于ES来说，通常来说单节点cpu 4核起步，不同角色的节点对CPU的要求也不同。</span> </section> <h4 data-sourcepos="240:1-240:77" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">CPU优化的第一步：角色分离，合理分配节点角色与资源</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在 ElasticSearch 集群中，明确划分主节点（Master）、数据节点（Data）和协调节点（Coordinating）的角色。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">主节点（Master）主要负责集群的管理和元数据存储，其 CPU 负载主要来自于集群状态的维护和更新。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">数据节点（Data）负责存储和索引数据，在数据写入和查询操作时会占用较多 CPU 资源，这是CPU的大户。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">协调节点（Coordinating）负责接收用户查询请求并将其分发到合适的数据节点进行处理，也会消耗一定的 CPU 资源用于请求的转发和结果的合并。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个大型的ES集群中，大致的CPU配置如下：</span> </section> <ul data-sourcepos="256:1-263:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">主节点（Master）配置在性能稳定、CPU 核心数适中（如 4 - 8 核）的服务器上，专门用于处理集群管理任务；</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">数据节点（Data）则根据数据量和查询负载，分配具有较高 CPU 核心数（如 16 - 32 核）的服务器，以满足大量数据的索引和查询操作；</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">协调节点（Coordinating）可以选择 CPU 核心数相对较少（如 8 - 16 核）但网络性能良好的服务器，用于高效地分发和接收查询请求。</span> </section></li> </ul> <h4 data-sourcepos="264:1-264:68" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">CPU优化的第二步：对线程池进行监控和动态调优</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>ElasticSearch 线程池类型</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ElasticSearch 中有多种线程池，如搜索线程池（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">search</span></code> <span style="font-size: 15px;">）、索引线程池（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">index</span></code> <span style="font-size: 15px;">）、批量线程池（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">bulk</span></code> <span style="font-size: 15px;">）等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">每个线程池负责不同类型的操作，并且其大小会影响 CPU 的利用率。例如:</span> </section> <ul data-sourcepos="274:1-276:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">搜索线程池（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">search</span></code> <span style="font-size: 15px;">）大小决定了可以同时处理的搜索查询数量</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">索引线程池（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">index</span></code> <span style="font-size: 15px;">）大小则影响数据索引的并发处理能力。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>优化线程池大小</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">根据 CPU 核心数和集群的实际负载情况，合理调整线程池大小。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一般来说，线程池大小不应超过 CPU 核心数的两倍，以避免过多的线程竞争 CPU 资源导致性能下降。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个具有 16 核 CPU 的数据节点上，可以将搜索线程池大小设置为 30 左右，索引线程池大小设置为 20 左右。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">同时，可以通过监控工具（如 ElasticSearch 自带的监控 API 或者第三方监控工具）观察线程池的活跃线程数、队列长度等指标，根据这些指标动态调整线程池大小。</span> </section> <h4 data-sourcepos="289:1-289:60" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">使用 ElasticSearch 自带监控 API获取监控数据</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ElasticSearch 提供了丰富的 REST API 用于监控集群状态。可以通过访问</span> </section> <pre data-sourcepos="293:1-295:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">_cat/thread_pool<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">端点来获取线程池相关的信息。例如，发送一个 HTTP GET 请求到</span> </section> <pre data-sourcepos="299:1-301:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">http://&lt;es-node-ip&gt;:9200/_cat/thread_pool?v<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">v参数用于以详细格式返回结果），会返回类似如下内容：</span> </section> <pre data-sourcepos="307:1-324:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">pool_name active queue rejected<br>bulk 0 0 0<br>fetch 0 0 0<br>flush 0 0 0<br>generic 0 0 0<br>get 0 0 0<br>index 0 0 0<br>listener 0 0 0<br>management 0 0 0<br>percolate 0 0 0<br>refresh 0 0 0<br>search 0 0 0<br>snapshot 0 0 0<br>suggest 0 0 0<br>war m 0 0 0<br>write 0 0 0<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">其中，</span> </section> <ul data-sourcepos="328:1-331:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">active</span></code> <span style="font-size: 15px;">列表示当前活跃的线程数，</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">queue</span></code> <span style="font-size: 15px;">列表示线程池队列中的任务数，</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">rejected</span></code> <span style="font-size: 15px;">列表示被拒绝的任务数。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过这些核心指标，可以解每个线程池的负载情况。</span> </section> <h4 data-sourcepos="334:1-334:64" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">动态调整线程池大小（通过 API 或配置文件）</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果发现 线程池负载高，或者 `queue 线程池队列中的任务数 积压多，可以进行动态调整线程池大小调整</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在 ElasticSearch 中，可以通过更新集群设置来动态调整线程池大小，可以使用下面的API，</span> </section> <pre data-sourcepos="340:1-342:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">_cluster/settings<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">发送一个 PUT类型的settings 请求，例如：</span> </section> <pre data-sourcepos="350:1-356:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">{<br> <span style="color: rgb(1, 86, 146);">"persistent"</span>: {<br> <span style="color: rgb(1, 86, 146);">"thread_pool.search.size"</span>: <span style="color: rgb(183, 85, 1);">30</span><br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这个请求将搜索线程池（search）的大小调整为 30。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">需要注意的是，这种调整方式需要谨慎操作，并且最好在低峰期进行，因为不当的调整可能会对集群性能产生暂时的负面影响。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">另外，也可以通过修改elasticsearch.yml配置文件来调整线程池大小，不过这种方式需要重启集群才能生效。</span> </section> <h4 data-sourcepos="364:1-364:27" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">CPU 的优化建议：</span></h4> <ul data-sourcepos="366:1-375:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">确保服务器有足够的CPU核心，并合理调整Elasticsearch的线程池配置，可以提高系统的并发处理能力。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">推荐使用多核处理器（至少8核及以上）来提升处理能力。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">避免虚拟化过度使用，特别是在生产环境中，因为这可能会引入不必要的性能开销。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">对线程池进行监控和动态调优， 防止线程任务积压</span> </section></li> </ul> <h3 data-sourcepos="376:1-376:51" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">1.3 为ES进行 硬件层面的 网络优化：</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><br></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">建议为ES 使用 低延迟的网络 ：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES是天生自带分布式属性的，并且ES的分布式系统是基于对等网络的，节点与节点之间的通信十分的频繁，延迟对于ES的用户体验是致命的，所以对于ES来说，低延迟的网络是非常有必要的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">高延迟的，跨多个数据中心的ES集群方案，是不太可取的，</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">虽然，ES可以容忍集群跨多个机房，可以有多个内网环境，支持跨AZ部署，但是不建议多个机房跨地域构建集群，</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><span style="color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;font-size: 15px;letter-spacing: normal;text-align: start;caret-color: rgb(36, 41, 46);">多个机房跨地域，&nbsp;</span>一旦发生了网络故障，集群可能直接雪崩，即使能够保证服务正常运行，维护这样（跨地域单个集群）的集群带来的额外成本可能远小于它带来的额外收益。</span> </section> <h2 data-sourcepos="394:1-394:26" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">2 系统层面的优化</span></h2> <h3 data-sourcepos="396:1-396:60" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">2.1 为ES进行 系统层面的 文件句柄数 调优</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">修改ES启动用户可使用的系统文件句柄数，以适应Elasticsearch的需求。</span> </section> <h4 data-sourcepos="400:1-400:79" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">Elasticsearch 需要大量 的文件句柄数， 主要 两大 原因：</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>第一：ES集群有大量的TCP /http 连接需求</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>大量的节点间通信连接</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch 集群中节点之间， 需要进行频繁的通信。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如数据同步、状态协调等，这会建立大量的 TCP 连接，每个 TCP 连接都需要占用一个文件句柄1。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>ES集群有大量的客户端连接</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当有大量的 HTTP 客户端连接到 Elasticsearch 集群进行数据读写操作时，也会消耗大量的文件句柄。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果文件句柄数不足，可能导致新的连接无法建立，客户端出现连接超时等问题 。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>第二：ES集群有大量的索引与数据操作</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>ES集群有大量的索引文件操作</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch 在处理索引时，会涉及到大量的文件操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在索引创建、更新、删除过程中，需要打开和操作多个索引文件，包括索引元数据文件、数据文件、段文件等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">每个文件都需要一个文件句柄，如果文件句柄数受限，可能会导致索引操作失败或异常。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>ES集群有大量的数据段合并</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">为了提高查询性能，Elasticsearch 会在后台进行数据段合并操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在合并过程中，需要同时打开多个段文件进行读写，如果文件句柄数不足，会影响段合并的效率，甚至导致合并失败 。</span> </section> <h4 data-sourcepos="432:1-432:77" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">问题：文件句柄数的 系统限制，满足不了es的句柄需求</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>系统默认文件句柄数限制过低</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">许多 Linux 发行版默认每个进程允许的文件句柄数通常只有 1024 或更低，这对于处理大量数据和高并发请求的 Elasticsearch 来说远远不够。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>避免资源耗尽风险</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果不调整文件句柄数，当 Elasticsearch 的文件句柄使用达到系统默认限制时，继续请求打开文件就会失败，可能导致数据丢失、节点故障等严重问题 。</span> </section> <h4 data-sourcepos="446:1-446:61" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">调优：如何解除 文件句柄数的 系统限制？</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>step1：查看当前系统文件句柄数限制</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">ulimit -n</span></code> <span style="font-size: 15px;">命令查看当前文件描述符限制，默认通常是 1024，对于 ES 来说远远不够。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>step2：修改文件句柄数</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>临时修改文件句柄数</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以 通过</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">ulimit -n 65536</span></code> <span style="font-size: 15px;">命令将当前会话的文件描述符限制临时设置为 65536 。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>永久修改文件句柄数</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">vim /etc/security/limits.conf</span></code> <span style="font-size: 15px;">命令编辑该limits.conf 文件，添加或修改以下配置项：</span> </section> <pre data-sourcepos="464:1-467:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">* soft nofile 65536<br>* hard nofile 65536<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">上述配置中，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">*</span></code> <span style="font-size: 15px;">表示匹配所有用户，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">soft</span></code> <span style="font-size: 15px;">为警告值，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">hard</span></code> <span style="font-size: 15px;">为最大值。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>检查配置是否生效</strong>：重新登录系统后，使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">ulimit -n</span></code> <span style="font-size: 15px;">命令查看文件句柄数限制是否已生效</span> </section> <h3 data-sourcepos="479:1-479:49" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">2.2 为ES进行 系统层面的 swap 调优</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这里就是简单粗暴，关掉swap。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">内存交换 到磁盘对服务器性能来说是 致命 的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果内存交换到磁盘上，性能就会急剧下降， 一个 100 微秒的操作可能变成 10 毫秒，下降100倍以上。想想那么多 10 微秒的操作时延累加起来。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">不难看出 swapping 对于性能是多么可怕。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">所以，Elasticsearch建议禁用Swap分区，因为当物理内存不足时，操作系统会将一些内存页交换到磁盘上，这会导致性能急剧下降。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在Linux系统中， 用以下命令关掉swap:</span> </section> <pre data-sourcepos="497:1-499:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">sudo swapoff -a<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在Linux系统中使用ES，最好可以通过修改</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">/etc/sysctl.conf</span></code> <span style="font-size: 15px;">文件来禁用Swap分区：</span> </section> <pre data-sourcepos="507:1-510:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;"># 在文件末尾添加以下行<br>vm.swappiness=<span style="color: rgb(183, 85, 1);">1</span><br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">然后运行</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">sudo sysctl -p</span></code> <span style="font-size: 15px;">使配置生效。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>除了 关掉swap ，还可以 使用 memlock ，为ES 锁定的物理内存。</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">memlock 是一个与内存锁定相关的系统资源限制。它用于指定一个进程能够锁定的物理内存（RAM）的大小。内存锁定意味着将进程使用的内存固定在物理内存中，防止操作系统将其交换（swap）到磁盘上的虚拟内存（磁盘交换空间）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于像 Elasticsearch 这样的应用程序，将内存锁定是很重要的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">因为 Elasticsearch 大量使用内存来缓存索引数据，如果这些数据被交换到磁盘，会导致查询性能大幅下降。内存锁定可以确保关键的内存区域（如索引缓存）始终在物理内存中，提高数据访问速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以通过设置Elasticsearch的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">bootstrap.memory_lock</span></code> <span style="font-size: 15px;">选项来尝试锁定JVM内存，防止其被交换到磁盘上， 在 配置文件elasticsearch.yml中添加以下行</span> </section> <pre data-sourcepos="526:1-529:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">bootstrap.memory_lock: <span style="color: rgb(183, 85, 1);">true</span><br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>注意</strong>：这需要用户 有</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">memlock</span></code> <span style="font-size: 15px;">权限。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以通过</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">ulimit -l</span></code> <span style="font-size: 15px;">命令查看当前用户的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">memlock</span></code> <span style="font-size: 15px;">限制，并通过</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">ulimit -l unlimited</span></code> <span style="font-size: 15px;">命令设置无限制（但这通常需要root权限）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在生产环境中，更推荐的方式是通过修改</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">/etc/security/limits.conf</span></code> <span style="font-size: 15px;">文件来永久设置 memlock 限制。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过修改</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">/etc/security/limits.conf</span></code> <span style="font-size: 15px;">文件来永久设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">memlock</span></code> <span style="font-size: 15px;">限制，可按以下步骤进行操作：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">用文本编辑器打开</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">/etc/security/limits.conf</span></code> <span style="font-size: 15px;">文件， 在文件末尾添加或修改以下内容 ：</span> </section> <pre data-sourcepos="543:1-546:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">* soft memlock [期望的软限制大小]<br>* hard memlock [期望的硬限制大小]<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，如果想将</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">memlock</span></code> <span style="font-size: 15px;">限制设置为 8GB（以字节为单位），则可写成：</span> </section> <pre data-sourcepos="554:1-557:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">* soft memlock 8388608<br>* hard memlock 8388608<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这里的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">*</span></code> <span style="font-size: 15px;">表示匹配所有用户，你也可以根据实际情况指定具体的用户，如</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">elastic soft memlock 8388608</span></code> <span style="font-size: 15px;">和</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">elastic hard memlock 8388608</span></code> <span style="font-size: 15px;">，只对</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">elastic</span></code> <span style="font-size: 15px;">用户生效 。</span> </section> <h2 data-sourcepos="569:1-569:25" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">3 JVM层面的优化</span></h2> <h2 data-sourcepos="573:1-573:52" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">3.1 为ES进行 JVM层面的 JVM堆大小调优</span></h2> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch是运行在JVM上的，对其做JVM参数调优至关重要。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">最常见的调优是Java内存的分配。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">下面是JVM的内存模型，具体每块的作用，不在这里阐述。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <img class="rich_pages wxw-img" data-imgfileid="100020697" data-ratio="0.5806451612903226" src="/upload/48a318301da6e990f78479f26c68778e.png" data-w="744" style="box-sizing: content-box;border-style: none;background-color: rgb(255, 255, 255);"> </section> <h3 data-sourcepos="583:1-583:58" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">新生代和老年代分配的内存比例给多大？</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Jvm内存分为新生代和老年代。</span> </section> <ul data-sourcepos="587:1-602:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">新生代（或者伊甸园）</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">新实例化的对象分配的空间。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">新生代空间通常都非常小，一般在 100 MB–500 MB。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">新生代也包含两个 幸存 空间。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">老年代</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">较老的对象存储的空间。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">这些对象预计将长期留存并持续上很长一段时间。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">老生代通常比新生代大很多。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">新生代、老生代的垃圾回收都有一个阶段会“stop the world”。给新生代和老年代分配多大的内存呢？他们的比例是多少呢？</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一般来说，老年代和新生代的内存比例为2：1是比较合适的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">比如给堆内存分配3G，则新生代分配1G，其余都给老年代。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在ElasticSearce的配置文件jvm.options文件配置：</span> </section> <pre data-sourcepos="611:1-615:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">-Xms3g //配置堆初始化大小<br>-Xmx3g //配置堆的最大内存<br>-Xmn1g //配置新生代内存。<br></span> </section></pre> <h3 data-sourcepos="617:1-617:51" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">一个Elasticesearch节点的两个内存大户</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">内存对于 Elasticsearch 来说绝对是重要的，它可以被许多内存数据结构使用来提供更快的操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一台机器装一个Elasticesearch节点，我们应该怎么分配机器的内存呢？</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">官方给出了解决方案，把一半（少于）的内存分配给Luence，另外的内存分配给ElasticSearch.</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">是一个Elasticesearch内存大户，底层的 Luence 另一个内存大户：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>第一个 内存大户Elasticsearch 缓存</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">（1） 文档 缓存数据</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">最重要的是缓存索引数据。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">首先 是 文档数据缓存。 当执行查询操作时，经常访问的文档数据会被缓存在堆内存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如：一个电商商品搜索系统使用 Elasticsearch，用户频繁搜索的热门商品信息（如商品名称、价格、品牌等）会被缓存，这样在后续查询时可以快速从内存中获取，减少磁盘 I/O 操作，提高查询响应速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">其次 是过滤器缓存， 也存储在堆内存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当使用过滤器进行查询（如筛选出特定品牌的商品）时，过滤器的结果会被缓存，下次遇到相同的过滤条件时可以直接利用缓存结果，提高过滤效率。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">（2） 索引缓冲区</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在数据写入索引的过程中，会有一个索引缓冲区（Index Buffer）在堆内存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">新的数据首先进入这个缓冲区，当缓冲区满或者达到一定的阈值时，数据会被刷新到磁盘上的索引文件中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">（3） 查询相关数据结构</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">存储查询执行过程中的临时数据结构，如用于排序的字段值集合。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对商品按照价格进行排序的查询，排序过程中的价格数据会暂存在堆内存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">还有聚合操作的中间结果，比如在统计不同品牌商品数量的聚合查询中，每个品牌的计数中间结果会存储在堆内存。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">（4） 集群状态 缓存</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch 主节点需要在堆内存中存储集群状态信息，包括索引元数据（如索引名称、分片数量、副本数量等）、节点信息（如节点角色、节点健康状态等）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这些信息对于集群的管理和协调至关重要，如在进行分片分配、故障转移等操作时会使用到这些信息。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>第二个 内存大户 Lucene 非堆内存（off - heap）</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">（1） 索引文件数据</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Lucene 是 Elasticsearch 的底层搜索引擎库，大部分索引数据实际上存储在非堆内存（磁盘文件）中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">索引文件包含了倒排索引，倒排索引是 Lucene 的核心数据结构。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对于文档中的每个单词，倒排索引记录了包含该单词的文档列表。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在文本搜索场景下，如新闻网站的文章搜索，文章中的词汇和对应的文档编号存储在非堆内存的索引文件中，用于快速定位包含特定词汇的文章。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">（2） 段文件（Segment Files）存储在非堆内存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Lucene 会将索引划分为多个段，每个段是一个独立的索引部分。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这些段文件包含了文档数据、词向量等信息，并且会根据一定的策略进行合并和优化。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在数据更新过程中，新的数据可能会形成新的段，然后通过段合并操作来优化索引结构，这些段文件存储在非堆内存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">（3）文件系统缓存</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">操作系统会对磁盘上的索引文件进行缓存，这部分缓存数据虽然不在 Java 堆内存中， 也是（off - heap）非堆内存的一部分。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当 Lucene 读取索引文件时，操作系统会将最近访问的文件块缓存在内存中，提高文件的访问速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，频繁被查询的索引段文件部分会被操作系统缓存，下次访问时可以直接从缓存中读取，减少磁盘读取时间。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Lucene 使用 内存映射文件（Memory - Mapped Files）数据，将磁盘上的索引文件映射到内存中，使得文件访问就像访问内存一样高效。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这部分内存映射的数据也属于非堆内存。通过这种方式，Lucene 可以直接在内存中操作索引文件的部分内容，而不需要进行大量的文件 I/O 操作，提高了索引数据的读写效率。</span> </section> <h3 data-sourcepos="697:1-697:48" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">该分配多大的内存给Elasticesearch？</span></h3> <ul data-sourcepos="701:1-708:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">如果 把所有的内存都分配给 Elasticsearch 的堆内存，那将不会有剩余的内存交给 Lucene。这将严重地影响全文检索的性能。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">标准的建议是把 50％ 的可用内存作为 Elasticsearch 的堆内存，保留剩下的 50％，Lucene 会很乐意利用起余下的内存，也就是把 50％ 的可用内存作为 Lucene 的非堆内存（off - heap）。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">Lucene 被设计为可以利用操作系统底层机制来缓存内存数据结构，所以这部分Lucene 的非堆内存（off - heap），预留就OK了，不需要专门设置。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">所以，在使用Elasticesearch的时候，按照实际的系统内存大小，一般进行Elasticesearch堆大小进行对应的配置，从最小规格的8G（如果 os内存 16g） 到了中等规模16G内存（如果 os内存 32g），然后最大规格32G内存（如果 os内存&gt;64g）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">分配的建议如下：</span> </section> <ul data-sourcepos="713:1-720:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">如果 os内存 16g ，jvm 8G</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">如果 os内存 32g ，jvm 16g</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">如果 os内存&gt;64g ，jvm 32g</span> </section></li> </ul> <h4 data-sourcepos="721:1-721:23" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">Xms和Xmx设置：</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">合理设置堆内存大小， 一般建议将堆内存设置为物理内存的 50 - 70%。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过-Xms（初始堆大小）和-Xmx（最大堆大小）参数来设置初始堆大小、最大堆大小 相同。例如：</span> </section> <pre data-sourcepos="729:1-732:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">-Xms30g<br>-Xmx30g<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">确保</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">-Xms</span></code> <span style="font-size: 15px;">和</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">-Xmx</span></code> <span style="font-size: 15px;">的值相同，这样可以避免在堆内存扩展或收缩过程中产生性能开销。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">建议将JVM堆内存的初始大小(-Xms)和最大大小(-Xmx)设置为相同的值，以避免运行过程中动态调整堆内存大小，这是一个损耗性能的操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以通过修改</span> </section> <pre data-sourcepos="740:1-742:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">jvm.options<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">文件来调整堆内存的大小。例如，将堆内存设置为16GB的配置如下：</span> </section> <pre data-sourcepos="746:1-749:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">-Xms16g<br>-Xmx16g<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这里将堆内存的初始大小和最大大小都设置为16GB，你可以根据实际情况调整</span> </section> <h3 data-sourcepos="757:1-757:76" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">分配给Elasticesearch 最大内存应该小于 32766 mb（~31.99 gb）</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">为啥呢？ JVM 在内存小于 32 GB 的时候会采用一个内存对象指针压缩技术。</span> </section> <ul data-sourcepos="761:1-764:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">对于 32 位的系统，意味着堆内存大小最大为 4 GB。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">对于 64 位的系统， 可以使用更大的内存，但是 64 位的指针意味着更大的浪费，因为你的指针本身大了。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">更糟糕的是， 更大的指针在主内存和各级缓存（例如 LLC，L1 等）之间移动数据的时候，会占用更多的带宽。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">所以， 为了节省 64 位的指针 占用内存的大小， Java 使用一个叫作 内存指针压缩（compressed oops）的技术来解决这个问题。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">使用 内存指针压缩（compressed oops）的技术， 对象的指针不再表示对象在内存中的精确位置，而是表示 偏移量 。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这意味着 32 位的指针可以引用 40 亿个 对象 ， 而不是 40 亿个字节。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">从极端角度来说，假设一个对象一个字节，使用 内存指针压缩（compressed oops）的技术后， ， 32 位的指针表示对象的指针理解为，最大增长到 32 GB 的物理内存。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一旦你越过那个神奇的 ~32 GB 的边界，指针就会切回普通对象的指针。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">切回普通对象的64位指针，每个对象的指针都变长了，就会使用更多的 CPU 内存带宽，也就是说你实际上失去了更多的内存。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">事实上，哪怕是 40–50 GB 内存，和使用了内存对象指针压缩技术时候的 32 GB 内存的 效果是一样的， 10多个G白白的给普通对象的64位指针了 。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">总之：即便你有足够的内存，JVM 也尽量不要 超过 32 GB。因为它浪费了内存，降低了 CPU 的性能，还要让 GC 应对大内存。</span> </section> <h2 data-sourcepos="785:1-785:31" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">3.2 垃圾回收器的优化</span></h2> <h3 data-sourcepos="789:1-789:38" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">3.2.1：JDK8 + G1 垃圾回收器</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">很多公司生产环境使用的是老的es 6 系列版本（比较稳定），如</span> </section> <ul data-sourcepos="795:1-802:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>JDK</strong>：JDK1.8_171-b11 (64位)</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>ES集群</strong>：一般由3台16核32G的虚拟机部署 ES 集群，每个节点分配16G堆内存</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>ELK版本</strong>：6.3.0</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>垃圾回收器</strong>：ES 默认指定的老年代（CMS）+ 新生代（ParNew）</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>操作系统</strong>：CentOS Linux release 7.4.1708(Core)</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch 默认的垃圾回收器（ GC ）是 CMS。官方建议使用 CMS。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这个垃圾回收器可以和应用并行处理，以便它可以最小化停顿。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">然而，它有两个 stop-the-world 阶段，但是 CMS处理大内存也有点吃力。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>启用 G1 垃圾回收器</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在 Elasticsearch 的 jvm.options文件中，将默认的垃圾回收器设置为 G1。找到以下行（如果不存在可以添加）：</span> </section> <pre data-sourcepos="817:1-819:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">-XX:+UseG1GC<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这会告诉 JVM 使用 G1 垃圾回收器来管理内存。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>设置 G1堆内存 参数</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">根据堆内存大小来调整 G1 的相关参数。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">第一：确定 G1 垃圾回收器的堆内存区域（Region）大小：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">-XX:G1HeapRegionSize</span></code> <span style="font-size: 15px;">参数用于确定 G1 垃圾回收器的堆内存区域（Region）大小。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">其默认值会根据堆内存大小自动调整，但在某些情况下可能需要手动优化。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">计算公式为：</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">1M &lt;= G1HeapRegionSize &lt;= 32M</span></code> <span style="font-size: 15px;">，并且</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">堆内存大小 / G1HeapRegionSize = Region数量</span></code> <span style="font-size: 15px;">应该是</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">2^n</span></code> <span style="font-size: 15px;">。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对于 30GB 的堆内存，可以尝试将</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">-XX:G1HeapRegionSize</span></code> <span style="font-size: 15px;">设置为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">16M</span></code> <span style="font-size: 15px;">，这样</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Region</span></code> <span style="font-size: 15px;">数量为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">30 * 1024 / 16 = 1920</span></code> <span style="font-size: 15px;">（近似值），是一个比较合适的数值。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">第二：优化停顿时间相关参数</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">设置最大停顿时间目标，使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">-XX:MaxGCPauseMillis</span></code> <span style="font-size: 15px;">参数来控制 G1 垃圾回收器的最大停顿时间目标。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，设置为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">200</span></code> <span style="font-size: 15px;">表示希望垃圾回收的停顿时间尽量不超过 200 毫秒。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">但要注意，设置的值过小可能会导致 G1 频繁地进行垃圾回收以满足停顿时间要求，反而增加了总的垃圾回收时间。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">需要根据实际的业务场景和性能测试来调整这个参数。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">第三：调整混合垃圾回收（Mixed GC）的触发条件</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">G1 会在一定条件下触发混合垃圾回收，即同时回收年轻代和部分老年代。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以通过</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">-XX:InitiatingHeapOccupancyPercent</span></code> <span style="font-size: 15px;">参数来调整触发混合垃圾回收的堆内存占用比例。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，将其设置为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">45</span></code> <span style="font-size: 15px;">，表示当堆内存占用达到 45% 时，G1 开始考虑触发混合垃圾回收。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这个参数的调整需要综合考虑堆内存大小、对象分配速率和停顿时间要求等因素。</span> </section> <h3 data-sourcepos="869:1-869:38" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">3.2.2：jdk17 + ZGC垃圾回收器</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">45岁老架构师尼恩建议，如果条件允许，可以升级jdk版本，提升性能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">那为什么要升级到jdk17呢，主要是由于：</span> </section> <ul data-sourcepos="875:1-880:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">1、jdk17的g1垃圾回收期相较于jdk11，提升到15%左右。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">2、在jdk17中，有向量化支持，同时在字符串处理等方面做了一些优化，相对于集群的写入压力会小很多。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">3、在jdk17中，对于ZGC的适配做了一些加强，这个ZGC的垃圾回收器比G1稳定性和性能有很大的提升。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在考虑把jdk的版本升级到jdk17去，同时尽量把elasticsearch的版本更新到最新的版本去。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在完成了jdk17的升级之后，下一步考虑的就是配置ZGC垃圾回收器。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ZGC垃圾回收器是一个并发的、单代的、基于区域的、NUMA 感知的垃圾回收器，Stop-the-world 阶段仅限于根扫描，因此 GC 暂停时间不会随堆或 live set 的变大而增加。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">而且ZGC垃圾回收期在回收的过程中几乎是全程并发处理，如下图：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <img class="rich_pages wxw-img" data-imgfileid="100020699" data-ratio="0.5185185185185185" src="/upload/c39675951add45f35e6628c2e788da30.png" data-type="png" data-w="1080" style="box-sizing: content-box;border-style: none;background-color: rgb(255, 255, 255);"> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">所以对于短暂停顿的时间来说非常小，几乎可以忽略不计。下面我们来演示下配置ZGC的垃圾回收器方法。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">关于zgc的文章，尼恩后续写一个系统化的介绍， 帮助大家穿透zgc。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在elasticsearch的config目录下有一个jvm.options文件</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <img class="rich_pages wxw-img" data-imgfileid="100020700" data-ratio="0.11587982832618025" src="/upload/77f338e33f0dd1faad07cf478e965797.png" data-type="png" data-w="932" style="box-sizing: content-box;border-style: none;background-color: rgb(255, 255, 255);"> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">打开后，找到如下部分：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <img class="rich_pages wxw-img" data-imgfileid="100020698" data-ratio="0.29463171036204744" src="/upload/4404bfa4ebe368cc1b3c761424fc04a7.png" data-type="png" data-w="801" style="box-sizing: content-box;border-style: none;background-color: rgb(255, 255, 255);"> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">因为我们的jdk版本是jdk17，所以找到14-的地方把这里所有的东西注释掉，然后添加如下的代码：</span> </section> <pre data-sourcepos="909:1-912:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;"><span style="color: rgb(183, 85, 1);">14</span>-:-XX:+UseZGC<br><span style="color: rgb(183, 85, 1);">14</span>-:-XX:+UnlockExperimentalVMOptions<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">最后的样子如下图：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <img class="rich_pages wxw-img" data-imgfileid="100020701" data-ratio="0.37454545454545457" src="/upload/30fe6498d0d9cd6e61f979e6649e272c.png" data-type="png" data-w="825" style="box-sizing: content-box;border-style: none;background-color: rgb(255, 255, 255);"> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">此时我们就配制好了当前的elasticsearch的环境使用ZGC垃圾收集器。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">然后我们启动下elasticsearch，启动完成后在日志中查找下ZGC关键词，如下图：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <img class="rich_pages wxw-img" data-imgfileid="100020703" data-ratio="0.13425925925925927" src="/upload/6b8ccf40da363556e2d8f700b686d1e6.png" data-type="png" data-w="1080" style="box-sizing: content-box;border-style: none;background-color: rgb(255, 255, 255);"> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以看到启动的时候使用的是ZGC垃圾收集器。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">以上就是ZGC垃圾收集器的配置全过程。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">1、zgc垃圾收集器从jdk11版本开始支持，所以只要是jdk11及以上的版本都支持zgc垃圾收集器。 2、zgc垃圾收集器的由于是并行进行处理，所以生产环境中ZGC的垃圾收集停顿时间不超过10ms，可以放心大胆的使用起来。 3、使用ZGC是并行的进行垃圾回收，所以有时候会发生查询耗时增加的情况，可以在jvm.options文件中添加如下的参数：-XX:+UseDynamicNumberOfGCThreads 代表动态配置GC线程，降低CPU的使用量，这个是JDK17的新特性，在jdk17以下是没有这个参数的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">具体如下图：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <img class="rich_pages wxw-img" data-imgfileid="100020702" data-ratio="0.5011764705882353" src="/upload/e777bd7485cb179df847b4c2541b63b0.png" data-type="png" data-w="850" style="box-sizing: content-box;border-style: none;background-color: rgb(255, 255, 255);"> </section> <h3 data-sourcepos="957:1-957:38" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">Elasticsearch的内存优化总结</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch的内存配置是其性能优化中非常重要的一部分。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">以下是一些关键点：</span> </section> <ul data-sourcepos="966:1-974:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch默认将JVM堆内存设置为1GB或系统物理内存的一半，取较小值。但是这个默认值可能不适合所有场景，因此需要根据实际情况进行调整。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">官方建议堆内存大小不要超过32GB，因为当堆大小超过32GB时，JVM会使用64位指针，这会导致更多的内存消耗和额外的开销。如果需要更大的堆内存，可以考虑增加节点数量而不是单个节点的堆内存大小。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">堆内存应该合理分配给Elasticsearch进程，同时要确保操作系统和其他进程有足够的内存可用。通常，将机器总内存的50%到60%分配给Elasticsearch是合理的，剩下的内存留给操作系统缓存和其他服务。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">对垃圾回收器进行优化，可以使用zgc</span> </section></li> </ul> <h2 data-sourcepos="975:1-975:40" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">4 ElasticSearch集群层面的优化</span></h2> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Elasticsearch</span></code> <span style="font-size: 15px;">的核心架构，是围绕着节点（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Node</span></code> <span style="font-size: 15px;">）和集群（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Cluster</span></code> <span style="font-size: 15px;">）展开。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">节点是</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Elasticsearch</span></code> <span style="font-size: 15px;">的运行实例，每个节点都承担着特定的任务和职责。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">多个节点组成一个集群，通过分布式的方式存储和处理数据，实现了高可用性、可扩展性和高性能。</span> </section> <h2 data-sourcepos="987:1-987:33" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">4.1. Elasticsearch节点概述</span></h2> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <br> </section> <p style="text-align: center;"><img class="rich_pages wxw-img js_insertlocalimg" data-imgfileid="100020704" data-ratio="0.812962962962963" data-s="300,640" src="/upload/ed1fb709b6f6b1c88fe5671cddf8a5de.png" data-type="png" data-w="1080" style=""></p> <h3 data-sourcepos="991:1-991:31" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.1.1 节点的基本概念</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Elasticsearch</span></code> <span style="font-size: 15px;">中，<strong>节点（Node）</strong>&nbsp;是一个运行中的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Elasticsearch</span></code> <span style="font-size: 15px;">实例。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">它是集群的基本组成单元，负责<strong>存储数据</strong>、<strong>处理查询请求</strong>以及<strong>与其他节点进行通信</strong>。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">每个节点都有一个唯一的标识符，用于在集群中进行识别和通信。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <br> </section> <p style="text-align: center;"><img class="rich_pages wxw-img js_insertlocalimg" data-imgfileid="100020705" data-ratio="1" data-s="300,640" src="/upload/a81c28534c2b88accc66559cfe10d19d.png" data-type="png" data-w="1024" style=""></p> <h3 data-sourcepos="1001:1-1001:31" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.1.2 节点的类型划分</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Elasticsearch</span></code> <span style="font-size: 15px;">中的节点可以根据其<strong>功能和角色</strong>进行不同的划分，常见的节点类型包括：</span> </section> <h5 data-sourcepos="1005:1-1005:40" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">4.1.2.1 主节点（Master Node）</span></h5> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>主节点</strong>&nbsp;负责管理整个集群的状态，包括节点的加入和离开、索引的创建和删除、分片的分配和迁移等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">主节点并不负责处理数据的存储和查询，它的主要职责是维护集群的元数据和协调各个节点的工作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">以下是一个简单的示例，展示如何在Elasticsearch的配置文件中设置节点为主节点：</span> </section> <pre data-sourcepos="1013:1-1018:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;"><span style="color: rgb(1, 86, 146);">node.name:</span> <span style="color: rgb(84, 121, 13);">master-node</span><br><span style="color: rgb(1, 86, 146);">node.master:</span> <span style="color: rgb(183, 85, 1);">true</span><br><span style="color: rgb(1, 86, 146);">node.data:</span> <span style="color: rgb(183, 85, 1);">false</span><br><br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在上述配置中，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">node.master</span></code> <span style="font-size: 15px;">&nbsp;设置为&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">true</span></code> <span style="font-size: 15px;">&nbsp;表示该节点可以作为候选主节点，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">node.data</span></code> <span style="font-size: 15px;">&nbsp;设置为&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">false</span></code> <span style="font-size: 15px;">&nbsp;表示该节点不存储数据。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这里设置的是候选主节点，真正的主节点通过选举产生，<strong>45岁老架构师尼恩提示， es 集群只能有一个 真正主节点。</strong></span> </section> <h5 data-sourcepos="1024:1-1024:41" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">4.1.2.2 数据节点（Data Node）</span></h5> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>数据节点</strong>&nbsp;负责存储和检索数据。它是集群中最核心的节点类型，负责处理文档的索引和查询操作。数据节点会将数据按照一定的规则进行分片存储，以实现数据的分布式存储和并行处理。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">以下是一个设置数据节点的示例配置：</span> </section> <pre data-sourcepos="1030:1-1035:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;"><span style="color: rgb(1, 86, 146);">node.name:</span> <span style="color: rgb(84, 121, 13);">data-node</span><br><span style="color: rgb(1, 86, 146);">node.master:</span> <span style="color: rgb(183, 85, 1);">false</span><br><span style="color: rgb(1, 86, 146);">node.data:</span> <span style="color: rgb(183, 85, 1);">true</span><br><br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在这个配置中，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">node.master</span></code> <span style="font-size: 15px;">&nbsp;设置为&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">false</span></code> <span style="font-size: 15px;">&nbsp;表示该节点不作为主节点，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">node.data</span></code> <span style="font-size: 15px;">&nbsp;设置为&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">true</span></code> <span style="font-size: 15px;">&nbsp;表示该节点是数据节点。</span> </section> <h5 data-sourcepos="1039:1-1039:49" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">4.1.2.3 协调节点（Coordinating Node）</span></h5> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>协调节点</strong>&nbsp;负责<strong>接收客户端的查询请求</strong>，并<strong>将请求转发到合适的数据节点进行处理</strong>。协调节点会收集各个数据节点的查询结果，并进行合并和排序，最终将结果返回给客户端。协调节点本身不存储数据，它的主要作用是协调和管理查询请求的处理过程。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">以下是一个设置协调节点的示例配置：</span> </section> <pre data-sourcepos="1045:1-1052:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;"><span style="color: rgb(1, 86, 146);">node.name:</span> <span style="color: rgb(84, 121, 13);">coordinating-node</span><br><span style="color: rgb(1, 86, 146);">node.master:</span> <span style="color: rgb(183, 85, 1);">false</span><br><span style="color: rgb(1, 86, 146);">node.data:</span> <span style="color: rgb(183, 85, 1);">false</span><br><span style="color: rgb(1, 86, 146);">node.ingest:</span> <span style="color: rgb(183, 85, 1);">false</span><br><span style="color: rgb(1, 86, 146);">search.remote.connect:</span> <span style="color: rgb(183, 85, 1);">false</span><br><br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在这个配置中，除了将&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">node.master</span></code> <span style="font-size: 15px;">&nbsp;和&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">node.data</span></code> <span style="font-size: 15px;">&nbsp;设置为&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">false</span></code> <span style="font-size: 15px;">&nbsp;外，还将&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">node.ingest</span></code> <span style="font-size: 15px;">&nbsp;和&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">search.remote.connect</span></code> <span style="font-size: 15px;">&nbsp;设置为&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">false</span></code> <span style="font-size: 15px;">，以确保该节点只作为协调节点。</span> </section> <h3 data-sourcepos="1056:1-1056:31" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.1.3 节点的通信机制</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">Elasticsearch</span></code> <span style="font-size: 15px;">中的节点通过&nbsp;<strong>TCP</strong>&nbsp;协议进行通信。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">节点之间会建立连接，通过发送和接收消息来进行数据的传输和交互。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">节点之间的通信是基于请求-响应模型的，一个节点发送请求，另一个节点接收请求并返回响应。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">为了提高通信效率和可靠性，Elasticsearch采用了一些优化措施，例如：</span> </section> <ul data-sourcepos="1066:1-1075:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>连接池</strong>：节点会维护一个连接池，用于管理与其他节点的连接。连接池可以复用已建立的连接，减少连接的建立和销毁开销。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>消息压缩</strong>：在节点之间传输消息时，Elasticsearch会对消息进行压缩，以减少网络传输的数据量。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>异步通信</strong>：节点之间的通信采用异步方式，发送请求后可以继续执行其他操作，不需要等待响应返回。</span> </section></li> </ul> <h2 data-sourcepos="1076:1-1076:31" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">4.2 合理划分节点角色</span></h2> <h3 data-sourcepos="1082:1-1082:34" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.2.1 主节点（Master）：</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">主节点负责集群的管理和元数据存储，如索引的创建、删除、分片分配等操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">为了确保集群的稳定性，应该选择性能稳定、网络连接良好的节点作为主节点。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一般建议将主节点与数据节点分离，避免主节点因处理大量的数据存储和查询任务而负载过重。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个大规模的日志分析集群中，可以选择 3 - 5 个配置较高的节点作为master candidate 主节点候选节点，并且这些节点最好分布在不同的机架或数据中心，以防止单点故障。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">同时，合理设置最小主节点，通过minimum_master_nodes 完成。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">minimum_master_nodes 设置及其重要，为了防止集群脑裂，这个参数应该设置为法定个数就是 ( master 候选节点个数 / 2) + 1。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>minimum_master_nodes的定义与作用</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">minimum_master_nodes</span></code> <span style="font-size: 15px;">是 Elasticsearch 集群中的一个重要设置，用于定义在集群中可以正常选举主节点（Master）的最少候选主节点数量。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">minimum_master_nodes 的主要目的是防止集群出现 “脑裂（split - brain）” 现象。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">脑裂是指在集群中，由于网络分区或其他原因，导致多个节点分别认为自己是主节点，从而使集群分裂成多个独立的部分，这会导致数据不一致和集群的混乱。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">minimum_master_nodes</span></code> <span style="font-size: 15px;">，可以确保在选举主节点时，只有当足够数量的候选主节点能够相互通信时，才能成功选举出主节点，从而避免脑裂情况的发生。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>计算minimum_master_nodes的合适值</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">计算</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">minimum_master_nodes</span></code> <span style="font-size: 15px;">的公式为：</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">(master_eligible_nodes / 2) + 1</span></code> <span style="font-size: 15px;">，</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">其中</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">master_eligible_nodes</span></code> <span style="font-size: 15px;">是集群中具有主节点选举资格的节点数量。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个有 5 个具有主节点选举资格的节点的集群中，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">minimum_master_nodes</span></code> <span style="font-size: 15px;">的值应该是</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">(5 / 2) + 1 = 3</span></code> <span style="font-size: 15px;">（向上取整）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这意味着在进行主节点选举时，至少需要 3 个候选主节点能够相互通信，才能成功选举出主节点。</span> </section> <h3 data-sourcepos="1122:1-1122:44" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.2.2 数据节点（Data）的配置：</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">数据节点主要负责存储和索引数据，它们会占用大量的磁盘空间和 I/O 资源。根据数据量和查询负载来合理分配数据节点的数量。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果数据量增长迅速或者查询并发量较高，可以适当增加数据节点。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">同时，要注意数据节点的硬件配置，如配备足够的内存来缓存索引数据，使用高性能的磁盘（如 SSD）来提高数据读写速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">除此之前，数据节点（Data）的配置的配置，和数据规模有关。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">预估存储一个亿文档的 Elasticsearch 数据节点数量需要综合多方面因素考虑，以下是一些分析：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>第一，评估数据量与存储</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>单个文档大小估算</strong>：假设每个文档平均大小为 1KB，那么一亿个文档大约需要 100GB 的存储空间。如果考虑副本以及索引的额外开销，实际所需空间会更大。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>节点存储容量规划</strong>：通常情况下，一个数据节点的磁盘容量建议不超过其总容量的 70% 至 80%，以预留一定空间用于数据的临时写入和系统文件存储等。如果使用普通机械硬盘，每个节点的存储容量可以控制在 2TB 至 3TB 左右；如果是 SSD 硬盘，可根据其性能和成本适当增加存储容量，但也不建议超过单节点 5TB 左右的存储量。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>第二，规划分片与副本</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>分片数量</strong>：根据经验，分片大小保持在 10GB 至 50GB 之间较为合适。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于一亿个文档（ 100GB），若按每个分片 30GB 左右估算，大约需要 4 至 5 个主分片。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">同时，还需要考虑副本数量，通常副本数量可以设置为 1 至 2 个，以提高数据的可用性和查询性能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>节点承载分片数</strong>：每个数据节点能够承载的分片数量也有限制，官方一般建议每 GB 堆内存对应少于 20 个分片。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">其实，官方的这个建议，这个其实性能很低的。为什么呢 ？</span> </section> <ul data-sourcepos="1156:1-1161:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">按照官方建议，一个具有 32GB 堆内存的数据节点，最多可承载 640 个分片左右， 对应到的磁盘空间 640 *30G =19200G 。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">假设缓存16G，对19200G 的数据进行缓存，当然 这么少的内存，这么大规模的数据，这个内存的缓存的作用，可以说聊胜于无，没啥命中率的。</span> </section></li> </ul> <h3 data-sourcepos="1162:1-1162:31" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">如果估算数据节点数</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">怎么估算较好呢？</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">尼恩的经验， 可以按照 内存需求，反向计算节点数。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一般来说， 最好有磁盘空间的30%以上的内存空间，作为 堆内存。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当然，这个比例越高，性能越好。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">比如有磁盘空间的50%以上的内存空间， es的性能越高。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">那么， 一亿个文档（ 100GB），加速副本最少 200G， 最好有60个G的内存。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">假设一个节点 16G内存， 那么 需要 5个节点。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当然，如果有10个节点， 性能会更好。</span> </section> <h3 data-sourcepos="1182:1-1182:52" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.2.3 协调节点（Coordinating）的配置：</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">协调节点负责接收用户查询请求，并将请求分发到合适的数据节点进行处理，然后汇总结果返回给用户。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">协调节点需要有良好的网络和 CPU 性能，以快速地转发和处理查询。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在集群中，可以根据查询流量来设置协调节点的数量，一般可以按照一定的比例（如每 10 个数据节点配置 2 - 3 个协调节点）来配置。</span> </section> <h2 data-sourcepos="1198:1-1198:37" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">4.3 ES 数据节点的写入调优</span></h2> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES 数据写入具有一定的延时性，这是为了减少频繁的索引文件产生。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">另外，在搜索引擎的业务场景下，用户一般并不需要那么高的写入实时性。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">比如 在网站发布一条征婚信息，或者二手交易平台发布一个商品信息。其他人并不是马上能搜索到的，这其实也是正常的处理逻辑。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">默认情况下 ES 每秒生成一个 segment 文件，当达到一定阈值的时候 会执行merge，merge 过程发生在 JVM中，频繁的生成 Segmen 文件可能会导致频繁的触发 FGC，导致 OOM。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">为了避免避免这种情况，通常采取的手段是降低 segment 文件的生成频率，手段有两个</span> </section> <ul data-sourcepos="1206:1-1208:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">一个是 增加时间阈值，</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">另一个是增大 Buffer的空间阈值，因为缓冲区写满也会生成 Segment 文件。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">生产经常面临的写入可以分为两种情况：</span> </section> <ul data-sourcepos="1211:1-1217:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">高频低量：高频的创建或更新索引或文档一般发生在 处理 C 端业务的场景下。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">低频高量：一般情况为定期重建索引或批量更新文档数据。</span> </section></li> </ul> <h3 data-sourcepos="1218:1-1218:31" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.3.1 写入优化的目标</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">写入调优均，常常以提升写入吞吐量和并发能力为目标，而非提升写入实时性。</span> </section> <h3 data-sourcepos="1222:1-1222:38" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.3.2.增加 flush 时间间隔，</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">目的是减小数据写入磁盘的频率，减小磁盘IO频率。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在 Elasticsearch 中，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">操作主要是将索引缓冲区（Index Buffer）中的数据刷新到磁盘上的索引文件中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当新的数据写入 Elasticsearch 时，会先存储在内存的索引缓冲区中。</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">操作会将这些缓冲区中的数据持久化，以确保数据的安全性和可恢复性。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">同时，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">操作还会更新索引的相关元数据，如提交点（Commit Point）等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>通过配置文件修改flush时间间隔</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在 Elasticsearch 的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">elasticsearch.yml</span></code> <span style="font-size: 15px;">配置文件中，可以找到与</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">相关的参数。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">主要参数是</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">index.translog.flush_threshold_period</span></code> <span style="font-size: 15px;">，它用于控制</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">的时间间隔。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这个参数的默认值是</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">30s</span></code> <span style="font-size: 15px;">，表示每隔 30 秒会自动进行一次</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">修改时间间隔要增加flush时间间隔，例如将其设置为60s，可以在elasticsearch.yml配置文件中添加或修改以下内容：</span> </section> <pre data-sourcepos="1244:1-1246:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;"><span style="color: rgb(1, 86, 146);">index.translog.flush_threshold_period:</span> <span style="color: rgb(84, 121, 13);">60s</span><br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">修改完成后，保存配置文件并重启 Elasticsearch 服务，新的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">时间间隔就会生效。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">尼恩提示，增加</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">时间间隔意味着数据在内存缓冲区中停留的时间更长。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果在这段时间内发生服务器故障或 Elasticsearch 进程意外终止，可能会导致缓冲区中的数据丢失。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">因此，在增加</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">时间间隔的同时，需要权衡数据安全性和性能之间的关系。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果数据的可靠性要求极高，不建议过长地延长</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">flush</span></code> <span style="font-size: 15px;">时间间隔。</span> </section> <h3 data-sourcepos="1262:1-1262:46" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.3.3 增加refresh_interval的参数值</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在 Elasticsearch 中，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">refresh_interval</span></code> <span style="font-size: 15px;">是一个用于控制索引数据可见性的重要参数。它定义了索引数据被刷新（Refresh）的时间间隔。当新的数据写入索引后，并不是立即可以被搜索到，而是要等待索引被刷新。刷新操作会使新写入的数据对搜索可见。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">具体来说，Elasticsearch 会在内存中维护一个数据结构来存储索引数据，这个数据结构在刷新之前是不断更新的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">refresh 刷新操作， 会将这个内存中的数据结构更新到磁盘上的索引文件（倒排索引等）中，并且更新相关的搜索视图，使得新数据能够被查询到。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">增加</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">refresh_interval</span></code> <span style="font-size: 15px;">的参数值 目的，是减少segment文件的创建，减少segment的merge次数，merge是发生在jvm中的，有可能导致full GC，增加refresh会降低搜索的实时性。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES的 refresh 行为非常昂贵，并且在正在进行的索引活动时经常调用，会降低索引速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>refresh_interval的默认值和取值范围</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>默认值</strong>：默认的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">refresh_interval</span></code> <span style="font-size: 15px;">是</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">1s</span></code> <span style="font-size: 15px;">。这意味着每隔 1 秒，索引会被刷新一次，新写入的数据会在 1 秒后对搜索可见。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>取值范围</strong>：可以设置为一个时间值，如</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">-1</span></code> <span style="font-size: 15px;">（表示禁用自动刷新，需要手动刷新）、</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">30s</span></code> <span style="font-size: 15px;">、</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">1m</span></code> <span style="font-size: 15px;">等。时间单位可以是秒（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">s</span></code> <span style="font-size: 15px;">）、分钟（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">m</span></code> <span style="font-size: 15px;">）、小时（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">h</span></code> <span style="font-size: 15px;">）等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">默认情况下，Elasticsearch 每秒（1s ） 定期刷新索引，但仅在最近 30 秒内收到一个或多个搜索请求的索引上。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果没有搜索流量或搜索流量很少（例如每 5 分钟不到一个搜索请求）， 并且想要优化索引速度，这是最佳配置。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">此行为旨在在不执行搜索的默认情况下自动优化批量索引。建议显式配置此配置项，如 30秒。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如何修改refresh_interval？ 通过配置文件修改，在elasticsearch.yml配置文件中，可以通过添加或修改以下内容来设置refresh_interval</span> </section> <pre data-sourcepos="1294:1-1296:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;"><span style="color: rgb(1, 86, 146);">index.refresh_interval:</span> <span style="color: rgb(84, 121, 13);">30s</span><br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这会将所有新创建的索引的刷新间隔设置为 30 秒。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果只想对特定的现有索引进行设置，可以使用索引设置 API。发送一个 PUT 请求到</span> </section> <pre data-sourcepos="1302:1-1304:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">http://&lt;es - node - ip&gt;:9200/&lt;index - name&gt;/_settings<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">，其中&lt;index - name&gt;是要修改的索引名称。请求体如下：</span> </section> <pre data-sourcepos="1310:1-1316:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">{<br> <span style="color: rgb(1, 86, 146);">"index"</span>: {<br> <span style="color: rgb(1, 86, 146);">"refresh_interval"</span>: <span style="color: rgb(84, 121, 13);">"30s"</span><br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这种方式可以在不重启 Elasticsearch 的情况下，动态地修改特定索引的刷新间隔。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">延长</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">refresh_interval</span></code> <span style="font-size: 15px;">可以提高索引的写入性能。因为刷新操作是一个相对较重的操作，它涉及到内存数据结构到磁盘索引的更新以及搜索视图的更新。减少刷新频率可以减少磁盘 I/O 和 CPU 的消耗，使得更多的资源可以用于数据写入。例如，在大数据量的批量写入场景下，将</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">refresh_interval</span></code> <span style="font-size: 15px;">延长到 30 秒或更长，可以显著提高写入速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">但是，延长</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">refresh_interval</span></code> <span style="font-size: 15px;">会导致新数据对搜索可见的延迟增加。如果应用场景对数据的实时性要求很高，如实时搜索系统，可能需要谨慎考虑延长刷新间隔的影响。</span> </section> <h3 data-sourcepos="1324:1-1324:51" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.3.4 写入的时候， 增加Buffer大小<span style="display: none;line-height: 0px;">‍</span></span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">本质也是减小refresh的时间间隔，因为导致segment文件创建的原因不仅有时间阈值，还有buffer空间大小，写满了也会创建。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">默认最小值 48MB&lt; 默认值 JVM 空间的10% &lt; 默认最大无限制</span> </section> <h3 data-sourcepos="1330:1-1330:41" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.3.5 写入的时候， 关闭副本</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当需要单次写入大量数据的时候，建议关闭副本，暂停搜索服务，或选择在检索请求量谷值区间时间段来完成。</span> </section> <ul data-sourcepos="1334:1-1336:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">第一是减小读写之间的资源抢占，读写分离</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">第二，当检索请求数量很少的时候，可以减少甚至完全删除副本分片，关闭segment的自动创建以达到高效利用内存的目的，因为副本的存在会导致主从之间频繁的进行数据同步，大大增加服务器的资源占用。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">具体可通过则设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">index.number_of_replicas</span></code> <span style="font-size: 15px;">&nbsp;为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">0</span></code> <span style="font-size: 15px;">以加快索引速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">没有副本意味着丢失单个节点可能会导致数据丢失，因此数据保存在其他地方很重要，以便在出现问题时可以重试初始加载。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">初始加载完成后，可以设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">index.number_of_replicas</span></code> <span style="font-size: 15px;">改回其原始值。</span> </section> <h2 data-sourcepos="1347:1-1347:26" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">4.4 数据分片设置</span></h2> <h3 data-sourcepos="1351:1-1351:32" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.4.1 数据分片的预估</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES一旦创建好索引后，就无法调整分片的设置，而在ES中，一个分片实际上对应一个lucene 索引，而lucene索引的读写会占用很多的系统资源，因此，分片数不能设置过大；</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">所以，在创建索引时，合理配置分片数是非常重要的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一般来说，我们遵循一些原则：</span> </section> <ol data-sourcepos="1359:1-1376:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">控制每个分片占用的硬盘容量，不超过ES的最大JVM的堆空间设置（一般设置不超过32G，参加上文的JVM设置原则），</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">因此，如果索引的总容量在500G左右，那分片大小在16个左右即可。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">一般都设置分片数不超过节点数的3倍。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">考虑一下node数量，一般一个节点有时候就是一台物理机，如果分片数过多，大大超过了节点数，很可能会导致一个节点上存在多个分片，</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">一旦该节点故障，即使保持了1个以上的副本，同样有可能会导致数据丢失，集群无法恢复。</span> </section> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">所以， 一般都设置分片数不超过节点数的3倍。</span> </section></li> </ol> <h3 data-sourcepos="1377:1-1377:29" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.4.2 分片分配策略</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES使用数据分片（shard）来提高服务的可用性，将数据分散保存在不同的节点上以降低当单个节点发生故障时对数据完整性的影响，同时使用副本（repiica）来保证数据的完整性。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">关于分片的默认分配策略，在7.x之前，默认5个primary shard，每个primary shard默认分配一个replica，即5主1副，而7.x之后，默认1主1副</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES的 分片分配策略，大致如下：</span> </section> <ul data-sourcepos="1385:1-1389:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">ES在分配单个索引的分片时会将每个分片尽可能分配到更多的节点上。但是，实际情况取决于集群拥有的分片和索引的数量以及它们的大小，不一定总是能均匀地分布。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">Paimary只能在索引创建时配置数量，而replica可以在任何时间分配，并且primary支持读和写操作，而replica只支持客户端的读取操作，数据由es自动管理，从primary同步。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">ES不允许Primary和它的Replica放在同一个节点中，并且同一个节点不接受完全相同的两个Replica</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">同一个节点允许多个索引的分片同时存在。</span> </section></li> </ul> <h3 data-sourcepos="1390:1-1390:25" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.4.3 分片的数量</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>避免分片过多</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">大多数搜索会命中多个分片。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">每个分片在单个 CPU 线程上运行搜索。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">虽然分片可以运行多个并发搜索，但跨大量分片的</span> <span style="font-size: 15px;">搜索</span> <span style="font-size: 15px;">会耗尽节点的</span> <span style="font-size: 15px;">搜索线程池</span> <span style="font-size: 15px;">。这会导致低吞吐量和缓慢的搜索速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>分片越少越好</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">每个分片都使用内存和 CPU 资源。在大多数情况下，一小组大分片比许多小分片使用更少的资源。</span> </section> <h3 data-sourcepos="1404:1-1404:31" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">4.4.4 分片的大小决策</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>分片的合理容量</strong>：10GB-50GB。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">虽然不是硬性限制，但 10GB 到 50GB 之间的分片往往效果很好。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">根据网络和用例，也许可以使用更大的分片。在索引的生命周期管理中，一般设置50GB为单个索引的最大阈值。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>堆内存容量和分片数量的关联</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">小于20分片/每GB堆内存，一个节点可以容纳的分片数量与节点的堆内存成正比。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，一个拥有 30GB 堆内存的节点最多应该有 600 个分片。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果节点超过每 GB 20 个分片，考虑添加另一个节点。</span> </section> <h1 data-sourcepos="1424:1-1424:40" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">5 ElasticSearch 索引层面的优化</span></h1> <h3 data-sourcepos="1428:1-1428:37" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.1.&nbsp;避免使用 dynamic mapping</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">尽量避免使用 dynamic mapping，即不要让 Elasticsearch 自动根据新插入的数据来推测字段类型并创建映射。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">而是在创建索引时，提前明确地定义好每个字段的类型、属性（如是否存储、是否索引、分词器等）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这样可以避免因数据类型不一致导致的索引混乱和性能问题，同时也能更好地控制索引的结构和资源占用。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个用户信息索引中，明确指定用户的年龄字段为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">integer</span></code> <span style="font-size: 15px;">类型，姓名字段为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">text</span></code> <span style="font-size: 15px;">类型并设置合适的分词器，而不是依赖 Elasticsearch 自动判断。</span> </section> <h3 data-sourcepos="1440:1-1440:45" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.2. 合理设置 doc_values 和 fielddata</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>doc_values定义与概念</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">doc_values 是 Elasticsearch 中一种用于存储正排索引的数据结构。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">它是在索引构建阶段就被创建并存储在磁盘上的，主要用于快速地基于字段进行排序、聚合和脚本计算等操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当执行这些操作时，Elasticsearch 会从磁盘读取 doc_values 数据，而不需要重新解析原始的文档源（_source）数据，这大大提高了这些操作的效率。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>doc_values工作原理</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于每个文档中的字段，doc_values 会以一种列式存储的方式记录其数据。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对于一个包含 “姓名” 和 “年龄” 字段的文档集合，doc_values 会分别存储所有文档的 “姓名” 列和 “年龄” 列的数据。这种列式存储结构在进行排序和聚合操作时非常高效，因为它可以快速地遍历和处理同一列的数据。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当进行排序操作时，比如按照年龄从小到大排序，Elasticsearch 可以直接从 doc_values 中读取年龄列的数据，然后进行排序，而不需要逐个文档去解析获取年龄信息。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">同样，在进行聚合操作（如计算年龄的平均值、统计不同姓名的数量等）时，doc_values 提供了一种高效的数据访问方式，使得这些操作能够快速完成。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>doc_values适用场景和优势</strong></span> </section> <ul data-sourcepos="1464:1-1467:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>排序操作</strong>：当需要对字段进行排序时，doc_values 的优势明显。例如，在电商产品搜索中，按照价格、销量等字段排序，或者在日志分析中按照时间戳排序，doc_values 能够快速提供排序所需的数据，提高排序效率。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>聚合操作</strong>：对于统计计算（如求和、平均值、最大值、最小值等）和分组聚合（如统计不同类别产品的数量、不同时间段内日志的数量等），doc_values 是必不可少的。它允许 Elasticsearch 快速访问字段数据，而不需要在内存中重新构建数据结构。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>脚本计算</strong>：在使用脚本进行数据处理时，如果脚本涉及到对字段的访问和操作，doc_values 可以提供高效的数据获取方式，提高脚本执行的速度。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>关闭不需要的doc_values</strong>&nbsp;</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于不需要进行聚合操作的字段，在索引映射（Mapping）中设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">"doc_values": false</span></code> <span style="font-size: 15px;">，以节省磁盘空间和资源消耗，提高查询速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对于一些仅用于存储但不参与聚合计算的日志时间戳字段，如果不需要对时间戳进行统计分析等聚合操作，就可以关闭其 doc_values。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当创建一个新的索引时，可以在索引映射中明确指定哪些字段不需要 doc_values。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，假设要创建一个名my_index的索引，其中包含title（文章标题）和content（文章内容）两个字段，且content字段不需要 doc_values，因为这个字段主要用于全文搜索，很少用于排序或聚合等操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以使用以下的索引映射定义来关闭content字段的 doc_values：</span> </section> <pre data-sourcepos="1480:1-1496:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">PUT /my_index<br>{<br> <span style="color: rgb(1, 86, 146);">"mappings"</span>: {<br> <span style="color: rgb(1, 86, 146);">"properties"</span>: {<br> <span style="color: rgb(1, 86, 146);">"title"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"text"</span>,<br> <span style="color: rgb(1, 86, 146);">"doc_values"</span>: <span style="color: rgb(183, 85, 1);">true</span><br> },<br> <span style="color: rgb(1, 86, 146);">"content"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"text"</span>,<br> <span style="color: rgb(1, 86, 146);">"doc_values"</span>: <span style="color: rgb(183, 85, 1);">false</span><br> }<br> }<br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>fielddata定义与概念</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">fielddata 是一种在内存中缓存的数据结构，主要用于支持对文本类型字段（text 类型）的聚合操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">fielddata 与 doc_values 不同，fielddata 是在运行时（runtime）构建的，并且存储在 JVM 堆内存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">fielddata 的目的是为了弥补 text 类型字段在默认情况下无法进行聚合操作的不足。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>fielddata工作原理</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当对 text 类型字段进行聚合操作时，Elasticsearch 会将该字段的所有词项（terms）加载到 fielddata 缓存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对于一个包含产品描述的 text 字段，当需要统计不同关键词在产品描述中出现的频率时，Elasticsearch 会先将产品描述中的所有词汇加载到 fielddata 缓存中，然后进行统计计算。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">fielddata 的构建是一个相对昂贵的过程，它需要对文本进行分词处理，并将分词后的结果存储在内存中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">而且，由于它存储在 JVM 堆内存中，对于大量的文本数据进行聚合操作时，可能会导致内存占用过大，甚至出现内存溢出（OOM）的情况。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>谨慎使用 fielddata</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">谨慎使用 fielddata，因为它会占用大量的 JVM 堆内存空间。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果确实需要对某个字段进行聚合操作，且该字段数据量较大，考虑提前规划好 JVM 内存设置，防止出现 OutOfMemoryError（OOM）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">同时，尽量避免在高并发或大数据量聚合场景下频繁使用 fielddata。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对于一个电商商品搜索索引中的商品描述字段，如果只是偶尔进行简单的关键词搜索，而几乎不会对该字段进行聚合操作，那么就不要开启 fielddata。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>创建新索引时关闭 fielddata</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当创建新索引时，可以在映射中指定字段的属性来控制 fielddata 的开启或关闭。假设创建一个名为new_index</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">的索引，包含product_name（产品名称）和product_description（产品描述）两个字段，且不想为这些字段开启 fielddata（因为这些字段主要用于搜索，而非复杂的文本聚合），可以按照以下方式定义索引映射：</span> </section> <pre data-sourcepos="1538:1-1554:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">PUT /new_index<br>{<br> <span style="color: rgb(1, 86, 146);">"mappings"</span>: {<br> <span style="color: rgb(1, 86, 146);">"properties"</span>: {<br> <span style="color: rgb(1, 86, 146);">"product_name"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"text"</span>,<br> <span style="color: rgb(1, 86, 146);">"fielddata"</span>: <span style="color: rgb(183, 85, 1);">false</span><br> },<br> <span style="color: rgb(1, 86, 146);">"product_description"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"text"</span>,<br> <span style="color: rgb(1, 86, 146);">"fielddata"</span>: <span style="color: rgb(183, 85, 1);">false</span><br> }<br> }<br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>启用特定聚合功能但不使用 fielddata</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于一些文本字段，如果需要进行简单的聚合操作，如统计不同词条的文档计数（这是一种比较常见的聚合场景），可以考虑使用keyword类型来代替text类型。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对于一个category（产品类别）字段，如果只想统计不同类别产品的数量，将其定义为keyword类型而不是text类型，就不需要开启 fielddata 来进行聚合操作。</span> </section> <pre data-sourcepos="1564:1-1576:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">PUT /new_index<br>{<br> <span style="color: rgb(1, 86, 146);">"mappings"</span>: {<br> <span style="color: rgb(1, 86, 146);">"properties"</span>: {<br> <span style="color: rgb(1, 86, 146);">"category"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"keyword"</span>,<br> <span style="color: rgb(1, 86, 146);">"fielddata"</span>: <span style="color: rgb(183, 85, 1);">false</span><br> }<br> }<br> }<br>}<br></span> </section></pre> <h3 data-sourcepos="1580:1-1580:35" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.3. 优化 ignore_above 设置</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于字符串类型的字段，根据业务需求合理设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">ignore_above</span></code> <span style="font-size: 15px;">参数。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">该参数指定了字符串字段的最大长度，超过此长度的文本将被忽略。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">将其设置得越小越好，但要确保不会丢失重要的业务数据。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个商品评论索引中，如果业务只关心评论的前 100 个字符用于搜索和分析，那么可以将评论内容字段的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">ignore_above</span></code> <span style="font-size: 15px;">设置为 100，这样可以减少索引的大小和资源消耗，同时提高查询性能。</span> </section> <h3 data-sourcepos="1594:1-1594:30" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.4. 调整 _source 字段</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">include</span></code> <span style="font-size: 15px;">和</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">exclude</span></code> <span style="font-size: 15px;">参数来精细控制哪些字段需要存储在</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">只保留对业务查询和显示有必要的字段，避免存储过多不必要的字段数据，以节省磁盘空间和网络带宽。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个新闻文章索引中，如果业务只需要在搜索结果中展示文章标题、发布时间和正文的前 200 个字符，那么可以使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">"_source": {"includes": ["title", "publish_time", "body.substring(0,200)"]}</span></code> <span style="font-size: 15px;">来配置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段，排除其他不需要的字段。</span> </section> <h3 data-sourcepos="1606:1-1606:34" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.5. 谨慎使用 store 属性</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于一些需要频繁访问但又不想每次都从</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">中提取的字段，可以考虑使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">store</span></code> <span style="font-size: 15px;">属性为其开辟单独的存储空间。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这样在查询时可以直接从存储的字段中获取数据，而无需解析</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">，从而节省网络带宽和查询时间。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">但要注意，过多地使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">store</span></code> <span style="font-size: 15px;">属性会增加磁盘空间的占用，所以需要根据字段的访问频率和重要性进行权衡。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，对于一个经常需要在搜索结果中单独展示的商品图片 URL 字段，可以设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">"store": true</span></code> <span style="font-size: 15px;">，以便快速获取该字段的值。</span> </section> <h3 data-sourcepos="1620:1-1620:45" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.6. 关于禁用 _source 字段的考虑</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">虽然禁用_source字段可以节省大量磁盘空间，但在决定禁用之前，必须充分考虑其带来的后果：</span> </section> <ul data-sourcepos="1626:1-1631:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">由于</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">update</span></code> <span style="font-size: 15px;">、</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">update_by_query</span></code> <span style="font-size: 15px;">和</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">reindex</span></code> <span style="font-size: 15px;">操作都依赖于</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段，禁用后这些操作将不可用。如果业务可能需要对索引数据进行更新或重新索引，那么禁用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段可能会带来不便。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">禁用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段会导致高亮显示功能失效，因为高亮显示通常是基于</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">中的原始文本进行的。如果搜索结果需要高亮显示匹配的关键词，那么禁用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段就不适合。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">禁用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段会影响索引的容灾能力，因为在数据丢失或损坏的情况下，没有</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段将难以恢复原始数据。因此，除非对磁盘空间有极其严格的限制，并且确定上述功能在业务中几乎不会用到，否则不建议轻易禁用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">字段。</span> </section></li> </ul> <h3 data-sourcepos="1632:1-1632:26" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.7. 禁用 all 字段</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在 Elasticsearch 6.0 及以上版本中，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">all</span></code> <span style="font-size: 15px;">字段默认是关闭的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果使用的是早期版本，且确定不需要在搜索时从所有字段中进行模糊检索（即不使用未指定字段名的搜索），那么可以手动关闭</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">all</span></code> <span style="font-size: 15px;">字段，以节省磁盘空间和提高索引创建速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个结构化数据索引中，每个字段的用途都很明确，很少会进行跨所有字段的模糊搜索，此时关闭</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">all</span></code> <span style="font-size: 15px;">字段是一个优化选择。</span> </section> <h3 data-sourcepos="1644:1-1644:32" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.8. 关闭 Norms 字段</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于那些确定不需要用于计算文档相关性评分（如在过滤查询和聚合操作中使用的字段），在索引映射中设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">"norms": false</span></code> <span style="font-size: 15px;">。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这样可以节省大量的磁盘空间，尤其是对于那些包含大量文本的字段。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个日志索引中，对于日志级别、IP 地址等字段，这些字段通常只用于过滤查询而不需要计算评分，将其</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">norms</span></code> <span style="font-size: 15px;">字段关闭可以优化索引性能。</span> </section> <h3 data-sourcepos="1656:1-1656:53" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.9. 谨慎关闭 index_options（高端操作）</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">index_options</span></code> <span style="font-size: 15px;">参数控制着在索引创建过程中哪些信息会被添加到倒排索引文件中，如词频（TF）、文档频率（docCount）、位置（postion）、偏移量（offsets）等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">减少这些选项可以降低索引创建时的 CPU 占用率，但需要谨慎操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">因为在实际业务中，很难预先确定将来是否会用到这些信息。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">除非在创建索引之前，对业务需求有非常清晰的了解，并且确定某些信息绝对不会被使用，否则不建议随意关闭</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">index_options</span></code> <span style="font-size: 15px;">中的选项，以免影响后续的查询功能和准确性。</span> </section> <h3 data-sourcepos="1670:1-1670:37" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.10. 合理设置 enabled 属性</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于那些不需要进行查询操作的字段，在索引映射中设置</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">"enabled": false</span></code> <span style="font-size: 15px;">，以避免创建不必要的倒排索引，从而节省磁盘空间和资源消耗。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，在一个只用于数据存储和备份的索引中，某些仅用于记录内部标识或辅助信息的字段，如果不需要对其进行搜索查询，就可以将其设置为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">enabled</span></code> <span style="font-size: 15px;">为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">false</span></code> <span style="font-size: 15px;">。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在实施上述优化方案时，需要根据具体的业务需求、数据特点和硬件资源进行综合考虑和调整，并且在优化前后进行充分的性能测试，以确保优化措施能够有效地提高 Elasticsearch 索引的性能和资源利用率。同时，随着业务的发展和数据的变化，可能需要定期对索引进行重新评估和优化，以维持良好的性能表现。</span> </section> <h3 data-sourcepos="1686:1-1686:35" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">5.11.&nbsp;max_result_window参数</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">max_result_window是分页返回的最大数值，默认值为10000。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">max_result_window本身是对JVM的一种保护机制，通过设定一个合理的阈值，避免初学者分页查询时由于单页数据过大而导致OOM。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在很多业务场景中经常需要查询10000条以后的数据，当遇到不能查询10000条以后的数据的问题之后，网上的很多答案会告诉你可以通过放开这个参数的限制，将其配置为100万，甚至1000万就行。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">但是如果仅仅放开这个参数就行，那么这个参数限制的意义有何在呢？</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如果你不知道这个参数的意义，很可能导致的后果就是频繁的发生OOM而且很难找到原因，设置一个合理的大小是需要通过你的各项指标参数来衡量确定的，比如你用户量、数据量、物理内存的大小、分片的数量等等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过监控数据和分析各项指标从而确定一个最佳值，并非越大越好</span> </section> <h2 data-sourcepos="1704:1-1704:41" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">6 ElasticSearch查询层面的优化</span></h2> <h4 data-sourcepos="1710:1-1710:35" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">避免使用稀疏数据</span></h4> <ul data-sourcepos="1718:1-1721:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="line-height: 1.75em;"> <span style="caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;font-size: 15px;letter-spacing: normal;text-align: start;">查询调优</span> <br> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">索引时间精度优化</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">研究Filter的工作原理可以看出，它每次工作都是遍历整个索引的，所以时间粒度越大，对比越快，搜索时间越短，在不影响功能的情况下，时间精度越低越好，有时甚至牺牲一点精度也值得，当然最好的情况是根本不作时间限制。 es重新刷索引，增加冗余的时间字段，精确到天。带有时间范围的查询使用该字段进行查询</span> </section> <ul data-sourcepos="1725:1-1726:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">查询Fetch Source优化</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">业务查询语句获取的数据集比较大，并且从source中获取了非必须的字段，导致查询较慢。 举例：只需要从es中查询id这一个字段，却把所有字段查询了出来</span> </section> <ul data-sourcepos="1730:1-1731:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">‘数据预索引’查询优化</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">数据预索引查询优化是一种在Elasticsearch中通过预先处理和索引数据来提高查询性能的技术。这种方法的核心思想是识别查询中的常见模式，并根据这些模式来优化数据的索引方式，从而加快查询速度。</span> </section> <h3 data-sourcepos="1736:1-1736:55" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.1&nbsp;调整filter过滤顺序, 过滤优先原则</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">问题：</span> </section> <ul data-sourcepos="1740:1-1745:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">如果把全文查询（如</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">match</span></code> <span style="font-size: 15px;">）的条件放在了前面，导致命中不了缓存，导致查询变慢。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="margin-top: 16px;margin-bottom: 16px;line-height: 1.75em;"> <span style="font-size: 15px;">如果把过滤效果不明显的条件放在了前面，导致查询出大量不需要的数据，导致查询变慢。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在Elasticsearch中，查询优化的一个关键方面是正确使用过滤器（filters）和查询（queries）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">过滤器（如</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">、</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">range</span></code> <span style="font-size: 15px;">等）通常比全文查询（如</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">match</span></code> <span style="font-size: 15px;">） 更快，为啥呢？</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">因为 filters 可以被缓存，而全文查询（如</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">match</span></code> <span style="font-size: 15px;">）则不可以被缓存。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">因此，调整过滤器和查询的顺序，优先使用过滤器，可以显著提高查询性能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">以下是一个使用Elasticsearch查询DSL的例子，展示了如何通过调整</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">filter</span></code> <span style="font-size: 15px;">和</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">query</span></code> <span style="font-size: 15px;">的顺序来优化查询：</span> </section> <h4 data-sourcepos="1756:1-1756:23" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">未优化的查询</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">假设我们有一个商品索引，包含商品的名称（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">name</span></code> <span style="font-size: 15px;">）、价格（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">price</span></code> <span style="font-size: 15px;">）和分类（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">category</span></code> <span style="font-size: 15px;">）字段。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一个未优化的查询可能如下：</span> </section> <pre data-sourcepos="1764:1-1793:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">GET /products/_search<br>{<br> <span style="color: rgb(1, 86, 146);">"query"</span>: {<br> <span style="color: rgb(1, 86, 146);">"bool"</span>: {<br> <span style="color: rgb(1, 86, 146);">"must"</span>: [<br> {<br> <span style="color: rgb(1, 86, 146);">"match"</span>: {<br> <span style="color: rgb(1, 86, 146);">"name"</span>: <span style="color: rgb(84, 121, 13);">"apple"</span><br> }<br> }<br> ],<br> <span style="color: rgb(1, 86, 146);">"filter"</span>: [<br> {<br> <span style="color: rgb(1, 86, 146);">"term"</span>: {<br> <span style="color: rgb(1, 86, 146);">"category"</span>: <span style="color: rgb(84, 121, 13);">"fruit"</span><br> }<br> },<br> {<br> <span style="color: rgb(1, 86, 146);">"range"</span>: {<br> <span style="color: rgb(1, 86, 146);">"price"</span>: {<br> <span style="color: rgb(1, 86, 146);">"lte"</span>: <span style="color: rgb(183, 85, 1);">100</span><br> }<br> }<br> }<br> ]<br> }<br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在这个查询中，我们首先使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">match</span></code> <span style="font-size: 15px;">查询来搜索名称中包含“apple”的商品，然后使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">filter</span></code> <span style="font-size: 15px;">来过滤属于“fruit”分类且价格不超过100的商品。</span> </section> <h4 data-sourcepos="1797:1-1797:23" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">优化后的查询</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">如何优化这个查询？</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可以将</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">filter</span></code> <span style="font-size: 15px;">条件移到</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">must</span></code> <span style="font-size: 15px;">子句中，因为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">filter</span></code> <span style="font-size: 15px;">可以被缓存，这样可以减少需要检查的文档数量，从而提高查询效率：</span> </section> <pre data-sourcepos="1805:1-1838:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">GET /products/_search<br>{<br> <span style="color: rgb(1, 86, 146);">"query"</span>: {<br> <span style="color: rgb(1, 86, 146);">"bool"</span>: {<br> <span style="color: rgb(1, 86, 146);">"must"</span>: [<br> {<br> <span style="color: rgb(1, 86, 146);">"bool"</span>: {<br> <span style="color: rgb(1, 86, 146);">"filter"</span>: [<br> {<br> <span style="color: rgb(1, 86, 146);">"term"</span>: {<br> <span style="color: rgb(1, 86, 146);">"category"</span>: <span style="color: rgb(84, 121, 13);">"fruit"</span><br> }<br> },<br> {<br> <span style="color: rgb(1, 86, 146);">"range"</span>: {<br> <span style="color: rgb(1, 86, 146);">"price"</span>: {<br> <span style="color: rgb(1, 86, 146);">"lte"</span>: <span style="color: rgb(183, 85, 1);">100</span><br> }<br> }<br> }<br> ]<br> }<br> },<br> {<br> <span style="color: rgb(1, 86, 146);">"match"</span>: {<br> <span style="color: rgb(1, 86, 146);">"name"</span>: <span style="color: rgb(84, 121, 13);">"apple"</span><br> }<br> }<br> ]<br> }<br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在这个优化后的查询中，我们首先使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">bool</span></code> <span style="font-size: 15px;">查询的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">filter</span></code> <span style="font-size: 15px;">子句来过滤出属于“fruit”分类且价格不超过100的商品，然后再使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">match</span></code> <span style="font-size: 15px;">查询来搜索名称中包含“apple”的商品。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这样，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">match</span></code> <span style="font-size: 15px;">查询只需要在已经过滤过的文档集上运行，从而提高了查询效率。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过这种方式，我们可以确保使用过滤器来减少搜索空间，然后再应用全文查询，这样可以显著提高查询性能，尤其是在处理大量数据时。</span> </section> <h3 data-sourcepos="1850:1-1850:30" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.2 使用 Keyword 类型</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ES中，并非所有数值数据都应映射为数值字段数据类型，</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch为 数值数据提供了 查询优化 ，例如</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">integer</span></code> <span style="font-size: 15px;">&nbsp;、 long。 如果不需要范围查找，对于 term查询而言，keyword 比 integer 性能更好。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">假设一个电商产品索引，其中包含产品 ID（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">product_id</span></code> <span style="font-size: 15px;">）和产品价格（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">price</span></code> <span style="font-size: 15px;">）两个字段。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">产品 ID 是一个唯一标识每个产品的数字，产品价格是实际的价格数值。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">我们将分别比较使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">keyword</span></code> <span style="font-size: 15px;">和</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">integer</span></code> <span style="font-size: 15px;">类型来存储产品 ID 在</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询中的性能差异。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">索引创建与数据插入，使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">integer</span></code> <span style="font-size: 15px;">类型存储产品 ID 并插入数据</span> </section> <pre data-sourcepos="1864:1-1878:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">PUT /products_index_integer_id<br>{<br> <span style="color: rgb(1, 86, 146);">"mappings"</span>: {<br> <span style="color: rgb(1, 86, 146);">"properties"</span>: {<br> <span style="color: rgb(1, 86, 146);">"product_id"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"integer"</span><br> },<br> <span style="color: rgb(1, 86, 146);">"price"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"float"</span><br> }<br> }<br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">插入一些示例数据，假设有 10000 个产品，产品 ID 从 1 到 10000，价格随机生成在 10.0 到 1000.0 之间。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">尼恩这里省略了过程， 大家可以自己设计一个脚本试一下。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">keyword</span></code> <span style="font-size: 15px;">类型存储产品 ID 并插入数据，创建另一个索引 products_index_keyword_id，将产品 ID 字段定义为keyword类型，产品价格字段同样定义为float类型。</span> </section> <pre data-sourcepos="1886:1-1900:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">PUT /products_index_keyword_id<br>{<br> <span style="color: rgb(1, 86, 146);">"mappings"</span>: {<br> <span style="color: rgb(1, 86, 146);">"properties"</span>: {<br> <span style="color: rgb(1, 86, 146);">"product_id"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"keyword"</span><br> },<br> <span style="color: rgb(1, 86, 146);">"price"</span>: {<br> <span style="color: rgb(1, 86, 146);">"type"</span>: <span style="color: rgb(84, 121, 13);">"float"</span><br> }<br> }<br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">使用相同的方法插入 10000 个产品数据到这个索引中。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">然后去做 精确匹配（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询） 的对比测试：</span> </section> <pre data-sourcepos="1906:1-1912:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">"query": {<br> "term": {<br> "product_id": 5000<br> }<br> }<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在多次测试后，你可能会发现，对于</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询产品 ID，使用</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">keyword</span></code> <span style="font-size: 15px;">类型的查询，比int 类型的查询，性能更好。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这是因为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">keyword</span></code> <span style="font-size: 15px;">类型在存储和查询数据时，内部的数据结构更适合精确匹配（</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">integer</span></code> <span style="font-size: 15px;">类型在 Elasticsearch 中是作为数值类型存储的，虽然它也支持</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询，但在底层存储和检索机制上，对于这种精确匹配的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询，会有一些额外的处理开销，比如数值范围的校验等（尽管在这个例子中没有涉及范围查询）。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">而</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">keyword</span></code> <span style="font-size: 15px;">类型就像一个简单的字符串存储结构，当进行</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询时，可以直接通过字符串匹配的方式快速定位到对应的文档，减少了不必要的处理步骤，从而提高了查询性能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>keyword 和 integer 类型的两大不同：</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">第一个不同：<strong>keyword 和 integer 类型的 数据存储结构差异：</strong></span> </section> <ul data-sourcepos="1926:1-1927:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">keyword 类型：</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当一个字段被定义为</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">keyword</span></code> <span style="font-size: 15px;">类型时，Elasticsearch 会将其视为一个简单<strong>的字符串</strong>进行存储。在底层，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">keyword</span></code> <span style="font-size: 15px;">类型的数据存储类似于一个字典结构，其中每个唯一的字符串值都有一个对应的标识符。在进行</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询时，Elasticsearch 直接在这个字典结构中查找给定的字符串，就像在一个键值对集合中查找键一样，这种查找方式非常直接和高效。</span> </section> <ul data-sourcepos="1930:1-1931:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">integer 类型：</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">integer</span></code> <span style="font-size: 15px;">类型，Elasticsear<strong>ch 会将其作为数值进行存</strong>储。虽然它也支持</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询，但由于其数值属性，在存储和检索过程中会涉及一些额外的处理。例如，它可能会考虑数值的范围、排序规则等因素，即使在进行简单的</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询时，这些额外的机制也可能会增加查询的处理步骤。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>第二个不同：索引构建和查询优化机制不同：</strong></span> </section> <ul data-sourcepos="1936:1-1937:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">keyword 类型：</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">keyword</span></code> <span style="font-size: 15px;">类型在索引构建过程中主要关注的是字符串的精确匹配。它会为每个不同的字符串值构建一个简单而高效的索引结构，用于快速定位包含该字符串的文档。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询时，Elasticsearch 可以直接利用这个索引结构，通过字符串的哈希值或者其他快速查找方法来定位文档，减少了不必要的计算和比较。</span> </section> <ul data-sourcepos="1942:1-1943:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">integer 类型：</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对于</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">integer</span></code> <span style="font-size: 15px;">类型，索引构建过程可能会考虑到数值的各种特性，如大小排序、范围划分等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在进行</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询时，虽然最终也是要找到匹配的数值，但它可能需要经过一些额外的验证步骤，以确保查询的数值符合数值类型的规则。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，它可能会检查数值是否在合理的范围之内，这种额外的检查在</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">term</span></code> <span style="font-size: 15px;">查询这种只需要精确匹配的场景下会增加一定的开销。</span> </section> <h3 data-sourcepos="1952:1-1952:27" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.3 避免使用脚本</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Scripting是Elasticsearch支持的一种专门用于复杂场景下支持自定义编程的强大的脚本功能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">相对于 DSL 而言，脚本的性能更差，DSL能解决 80% 以上的查询需求，如非必须，尽量避免使用 Script</span> </section> <h3 data-sourcepos="1958:1-1958:38" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.4 避免单次召回大量数据</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">搜索引擎最擅长的事情是从海量数据中查询少量相关文档，而非单次检索大量文档。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">非常不建议动辄查询上万数据。如果有这样的需求，建议使用滚动查询</span> </section> <h3 data-sourcepos="1964:1-1964:33" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.5 避免单个文档过大</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">鉴于默认</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="color: rgb(0, 122, 255);text-decoration: underline;font-size: 15px;">http.max_content_length</span></code> <span style="font-size: 15px;">设置为 100MB，Elasticsearch 将拒绝索引任何大于该值的文档。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">您可能决定增加该特定设置，但 Lucene 仍然有大约 2GB 的限制。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">即使不考虑硬性限制，大型文档通常也不实用。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">大型文档对网络、内存使用和磁盘造成了更大的压力，即使对于不请求的搜索请求也是如此，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_source</span></code> <span style="font-size: 15px;">因为 Elasticsearch</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">_id</span></code> <span style="font-size: 15px;">在所有情况下都需要获取文档的文件系统缓存有效。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">对该文档进行索引可能会占用文档原始大小的倍数的内存量。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Proximity Search（例如短语查询）和高亮查询也变得更加昂贵，因为它们的成本直接取决于原始文档的大小。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">有时重新考虑信息单元应该是什么是有用的。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，您想让书籍可搜索的事实并不一定意味着文档应该包含整本书。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">使用章节甚至段落作为文档可能是一个更好的主意，然后在这些文档中拥有一个属性来标识它们属于哪本书。这不仅避免了大文档的问题，还使搜索体验更好。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，如果用户搜索两个单词</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">foo</span></code> <span style="font-size: 15px;">and&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">bar</span></code> <span style="font-size: 15px;">，则不同章节之间的匹配可能很差，而同一段落中的匹配可能很好。</span> </section> <h3 data-sourcepos="1986:1-1986:63" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.6 单次查询10条文档 好于 10次查询每次一条</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">批量请求将产生比单文档索引请求更好的性能。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">但是每次查询多少文档最佳，不同的集群最佳值可能不同，为了获得批量请求的最佳阈值，建议在具有单个分片的单个节点上运行基准测试。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">首先尝试一次索引 100 个文档，然后是 200 个，然后是 400 个等。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在每次基准测试运行中，批量请求中的文档数量翻倍。当索引速度开始趋于平稳时，就可以获得已达到数据批量请求的最佳大小。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在相同性能的情况下，当大量请求同时发送时，太大的批量请求可能会使集群承受内存压力，因此建议避免每个请求超过几十兆字节。</span> </section> <h3 data-sourcepos="1998:1-1998:47" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.7 避免 Nested 内嵌 和 Join 连接</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">很多人会忽略对 Elasticsearch 数据建模的重要性。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一个特别要注意的是，应避免Nested 内嵌。nested属于object类型的一种，是Elasticsearch中用于复杂类型对象数组的索引操作。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">Elasticsearch没有内部对象的概念，因此，ES在存储复杂类型的时候会把对象的复杂层次结果扁平化为一个键值对列表。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">另外一个特别要注意的是，应避免连接。Nested 可以使查询慢几倍，Join 会使查询慢数百倍。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">两种类型的使用场景应该是：Nested针对字段值为非基本数据类型的时候，而Join则用于 当子文档数量级非常大的时候。</span> </section> <h3 data-sourcepos="2016:1-2016:32" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.8 使用filter代替query</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">query和filter的主要区别在： filter是结果导向的而<strong>query</strong>是过程导向。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">query倾向于“当前文档和查询的语句的相关度”而filter倾向于“当前文档和查询的条件是不是相符”。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">即在查询过程中，query是要对查询的每个结果计算相关性得分的，而filter不会。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">另外filter有相应的缓存机制，可以提高查询效率。</span> </section> <h3 data-sourcepos="2026:1-2026:26" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.9 避免深度分页</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">避免单页数据过大，可以参考百度或者淘宝的做法。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">es提供两种解决方案 scroll search 和 search after。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">关于深度分页的详细原理，推荐阅读：尼恩的技术自由圈的博客</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">ElasticSearch 深度分页 （史上最全、面试必备）_elasticsearch深度分页-CSDN博客</span> </section> <h3 data-sourcepos="2038:1-2038:56" style="margin: 10px auto 5px;font-weight: bold;font-size: 20px;letter-spacing: 0.578px;white-space: normal;background-color: rgb(252, 252, 252);"><span style="margin-top: -1px;padding: 6px 20px 6px 10px;font-size: 17px;font-weight: normal;display: inline-block;line-height: 1.3;background-color: rgb(212, 224, 250);border-bottom-right-radius: 100px;color: rgb(30, 30, 30);">6.10 预索引： ‘数据预索引’查询优化</span></h3> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">利用查询中的模式来优化数据的索引方式。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如，如果所有文档都有一个</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">price</span></code> <span style="font-size: 15px;">字段，并且大多数查询 range 在固定的范围列表上运行聚合，可以通过将范围预先索引到索引中并使用聚合来加快聚合速度。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">以下是一个基于电商系统中商品销售数据统计的案例，来展示 ElasticSearch 的 数据预索引 的优化方案。</span> </section> <h4 data-sourcepos="2048:1-2048:17" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">案例背景</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">一个电商平台，平台上有大量的商品在售卖，每天都会产生众多的销售订单记录。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">我们希望通过 ElasticSearch 对这些销售数据进行分析，以便了解不同年龄段用户的购买行为、商品销量情况等信息。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">在业务需求中，经常需要按照用户的年龄范围来进行聚合查询，例如统计不同年龄区间（如 18 - 25 岁、26 - 35 岁、36 - 45 岁等）内购买各类商品的数量、金额等指标。</span> </section> <h4 data-sourcepos="2056:1-2056:29" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">优化前的查询方式</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>优化前的数据结构</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">销售订单数据以文档的形式存储在 ElasticSearch 索引中，每个文档代表一笔销售订单，包含了多个字段，比如</span> </section> <ul data-sourcepos="2062:1-2069:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">order_id</span></code> <span style="font-size: 15px;">（订单编号）</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">product_name</span></code> <span style="font-size: 15px;">（商品名称）</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">price</span></code> <span style="font-size: 15px;">（商品价格）</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">user_age</span></code> <span style="font-size: 15px;">（用户年龄）</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">purchase_date</span></code> <span style="font-size: 15px;">（购买日期）等。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>优化前的查询操作</strong>：</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当业务人员想要查看不同年龄区间的商品销售总额时，通常会使用如下的聚合查询语句：</span> </section> <pre data-sourcepos="2076:1-2108:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">{<br> <span style="color: rgb(1, 86, 146);">"size"</span>: <span style="color: rgb(183, 85, 1);">0</span>,<br> <span style="color: rgb(1, 86, 146);">"aggs"</span>: {<br> <span style="color: rgb(1, 86, 146);">"age_ranges"</span>: {<br> <span style="color: rgb(1, 86, 146);">"range"</span>: {<br> <span style="color: rgb(1, 86, 146);">"field"</span>: <span style="color: rgb(84, 121, 13);">"user_age"</span>,<br> <span style="color: rgb(1, 86, 146);">"ranges"</span>: [<br> {<br> <span style="color: rgb(1, 86, 146);">"from"</span>: <span style="color: rgb(183, 85, 1);">18</span>,<br> <span style="color: rgb(1, 86, 146);">"to"</span>: <span style="color: rgb(183, 85, 1);">25</span><br> },<br> {<br> <span style="color: rgb(1, 86, 146);">"from"</span>: <span style="color: rgb(183, 85, 1);">26</span>,<br> <span style="color: rgb(1, 86, 146);">"to"</span>: <span style="color: rgb(183, 85, 1);">35</span><br> },<br> {<br> <span style="color: rgb(1, 86, 146);">"from"</span>: <span style="color: rgb(183, 85, 1);">36</span>,<br> <span style="color: rgb(1, 86, 146);">"to"</span>: <span style="color: rgb(183, 85, 1);">45</span><br> }<br> ]<br> },<br> <span style="color: rgb(1, 86, 146);">"aggs"</span>: {<br> <span style="color: rgb(1, 86, 146);">"total_sales"</span>: {<br> <span style="color: rgb(1, 86, 146);">"sum"</span>: {<br> <span style="color: rgb(1, 86, 146);">"field"</span>: <span style="color: rgb(84, 121, 13);">"price"</span><br> }<br> }<br> }<br> }<br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">上述查询语句的含义是，按照&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">user_age</span></code> <span style="font-size: 15px;">&nbsp;字段进行范围划分，分成几个指定的年龄区间，</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">然后在每个年龄区间内对商品的价格进行求和聚合，以得到不同年龄区间的商品销售总额。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">可能每天有成千上万笔订单， 随着订单数据量的不断增大，这样频繁地执行范围聚合查询会对 ElasticSearch 集群的性能造成巨大压力</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">结果是：查询响应时间可能会逐渐变长，影响业务人员获取数据的效率。</span> </section> <h4 data-sourcepos="2124:1-2124:38" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">预索引数据查询优化方案</span></h4> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>预聚合数据设计</strong></span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">我们决定在数据写入 ElasticSearch 索引时，就提前对用户年龄进行预聚合处理。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">新增一个字段，比如叫 age_group</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">age_group 的值根据 user_age 字段预先划分好的区间来确定。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">例如：</span> </section> <ul data-sourcepos="2140:1-2145:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">当&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">user_age</span></code> <span style="font-size: 15px;">&nbsp;在 18 - 25 岁之间时，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">age_group</span></code> <span style="font-size: 15px;">&nbsp;的值设为&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">18-25</span></code> <span style="font-size: 15px;">。</span> </section></li> <li style="margin-top: 0.25em;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">当&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">user_age</span></code> <span style="font-size: 15px;">&nbsp;在 26 - 35 岁之间时，</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">age_group</span></code> <span style="font-size: 15px;">&nbsp;的值设为&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">26-35</span></code> <span style="font-size: 15px;">，</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">以此类推。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这样，在数据索引阶段，在将订单数据导入 ElasticSearch 时，添加一段逻辑来判断&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">user_age</span></code> <span style="font-size: 15px;">&nbsp;的值，并设置对应的&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">age_group</span></code> <span style="font-size: 15px;">&nbsp;值。</span> </section> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">比如，我们可以通过 Logstash（或者其他 ETL 工具）在将订单数据导入 ，示例 Logstash 配置片段如下：</span> </section> <pre data-sourcepos="2152:1-2165:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">filter {<br> <span style="color: rgb(1, 86, 146);">if</span> [user_age] &gt;= <span style="color: rgb(183, 85, 1);">18</span> &amp;&amp; [user_age] &lt;= <span style="color: rgb(183, 85, 1);">25</span> {<br> mutate {<br> add_field =&gt; { <span style="color: rgb(84, 121, 13);">"age_group"</span> =&gt; <span style="color: rgb(84, 121, 13);">"18-25"</span> }<br> }<br> } <span style="color: rgb(1, 86, 146);">else</span> <span style="color: rgb(1, 86, 146);">if</span> [user_age] &gt; <span style="color: rgb(183, 85, 1);">25</span> &amp;&amp; [user_age] &lt;= <span style="color: rgb(183, 85, 1);">35</span> {<br> mutate {<br> add_field =&gt; { <span style="color: rgb(84, 121, 13);">"age_group"</span> =&gt; <span style="color: rgb(84, 121, 13);">"26-35"</span> }<br> }<br> }<br> <span style="color: rgb(101, 110, 119);"># 其他年龄区间的判断逻辑依次添加</span><br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;"><strong>优化后的查询操作</strong>： 现在，当业务人员想要查询不同年龄区间的商品销售总额时，就可以使用基于&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">age_group</span></code> <span style="font-size: 15px;">&nbsp;字段的更简单高效的聚合查询语句：</span> </section> <pre data-sourcepos="2174:1-2193:3" style="letter-spacing: normal;text-align: start;font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;margin-bottom: 16px;overflow-wrap: normal;overflow: auto;line-height: 1.45;border-radius: 4px;border-width: thin;border-style: solid;border-color: rgb(224, 224, 224);caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);"> <section style="display: block;overflow-x: auto;padding: 16px;color: rgb(47, 51, 55);background-color: rgb(246, 246, 246);font-family: Menlo, ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Courier, &quot;Courier New&quot;, Monaco, monospace, system-ui, ui-serif, ui-rounded;font-size: 13.6px;word-break: normal;border-radius: 3px;overflow-wrap: normal;line-height: 1.75em;"> <span style="font-size: 15px;">{<br> <span style="color: rgb(1, 86, 146);">"size"</span>: <span style="color: rgb(183, 85, 1);">0</span>,<br> <span style="color: rgb(1, 86, 146);">"aggs"</span>: {<br> <span style="color: rgb(1, 86, 146);">"age_groups"</span>: {<br> <span style="color: rgb(1, 86, 146);">"terms"</span>: {<br> <span style="color: rgb(1, 86, 146);">"field"</span>: <span style="color: rgb(84, 121, 13);">"age_group"</span>,<br> <span style="color: rgb(1, 86, 146);">"size"</span>: <span style="color: rgb(183, 85, 1);">10</span><br> },<br> <span style="color: rgb(1, 86, 146);">"aggs"</span>: {<br> <span style="color: rgb(1, 86, 146);">"total_sales"</span>: {<br> <span style="color: rgb(1, 86, 146);">"sum"</span>: {<br> <span style="color: rgb(1, 86, 146);">"field"</span>: <span style="color: rgb(84, 121, 13);">"price"</span><br> }<br> }<br> }<br> }<br> }<br>}<br></span> </section></pre> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">这个查询语句通过对&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">age_group</span></code> <span style="font-size: 15px;">&nbsp;字段进行&nbsp;</span> <code style="font-family: ui-monospace, SFMono-Regular, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace;font-size: 13.6px;padding: 0.14em 0.3em;vertical-align: 5%;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(223, 226, 229);"><span style="font-size: 15px;">terms</span></code> <span style="font-size: 15px;">&nbsp;聚合（也就是按照不同的年龄分组值进行分组统计），然后在每个分组内对商品价格进行求和聚合，来获取不同年龄区间的销售总额。</span> </section> <h4 data-sourcepos="2201:1-2201:23" style="margin: 10px auto -1px;padding-left: 3px;font-weight: bold;font-size: 18px;letter-spacing: 0.578px;white-space: normal;border-left-width: 10px;border-left-style: solid;border-left-color: rgb(222, 235, 255);"><span style="margin-top: -1px;padding-top: 6px;padding-right: 5px;padding-left: 5px;font-size: 16px;display: inline-block;line-height: 1.1;">优化效果对比</span></h4> <ul data-sourcepos="2205:1-2211:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>性能提升</strong>： 在未进行预索引数据查询优化之前，对于千万级别的订单数据进行年龄范围聚合查询，平均响应时间可能在 5 - 10 秒左右，随着数据量继续增加，响应时间还会进一步拉长。而经过优化后，同样的数据量下，查询响应时间可以缩短到 1 - 2 秒以内，大大提高了业务人员获取数据进行分析的效率。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;"><strong>资源利用优化</strong>： 从 ElasticSearch 集群的资源角度来看，未优化时，复杂的范围聚合查询需要占用较多的 CPU 资源来进行实时的年龄区间划分和数据计算。优化后，由于是基于预聚合的字段进行简单的分组统计，对 CPU 的消耗大幅降低，同时也减少了内存中临时数据的占用，使得集群整体可以更高效地处理更多的查询请求，提升了集群的资源利用率。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过这个电商销售数据统计的案例可以清晰地看到，对于业务中频繁出现的特定范围聚合查询需求，采用预索引数据查询优化的方式，提前对相关数据进行处理和聚合，可以显著提升 ElasticSearch 的查询性能，更好地满足业务分析需求。</span> </section> <h2 data-sourcepos="2216:1-2216:33" style="font-weight: bold;font-size: 22px;margin: 10px auto 5px;border-top-width: 1px;border-top-style: solid;border-top-color: rgb(242, 242, 242);background-color: rgb(242, 242, 242);"><span style="margin-top: -1px;padding-top: 14px;padding-bottom: 14px;padding-right: 5px;padding-left: 5px;font-size: 17px;border-top: 4px solid rgb(33, 33, 34);display: inline-block;line-height: 1.5;font-weight: normal;background-color: rgb(30, 30, 30);border-bottom-right-radius: 100px;color: rgb(255, 255, 255);padding-right: 20px;padding-left: 10px;">7 职责分离，全面监控</span></h2> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">当某个单一角色所负责的业务，或某个单一节点无法满足其业务需要的时候，最好的策略就是 解耦和分离, 常见的操作如：</span> </section> <ul data-sourcepos="2220:1-2224:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">功能分离：本质上也利于服务的轻量化， 百度 搜索引擎的内部人员 聊过 ，百度内部就是基于 Elasticsearch 的源码做修改，删除对其业务不需要的代码，对 ES做轻量化处理</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">业务分离：最简单常见的场景，当我在全文检索服务的时候，应避免在服务期间去执行大量的聚合分析。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">读写分离：常见操作，不做过多解释</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">冷热分离：基于索引生命周期管理策略下的性能动态分配策略。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">另外， 为了更高的性能， 需要进行密切的、全面的指标监控：</span> </section> <ul data-sourcepos="2227:1-2231:0" style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;padding-left: 2em;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;" class="list-paddingleft-1"> <li style="font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">定期监控集群的健康状况和资源使用情况，以便及时做出调整。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">监控线程池的情况，进行在线的扩容</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">根据实际需求调整索引设置，比如分片数、副本数等。</span> </section></li> <li style="margin-top: 0.25em;font-size: 15px;"> <section style="line-height: 1.75em;"> <span style="font-size: 15px;">考虑使用冷热架构，将活跃数据放在高性能节点上，历史数据则可以迁移到成本更低、性能稍弱的节点上。</span> </section></li> </ul> <section style="font-size: 16px;letter-spacing: normal;text-align: start;white-space: normal;margin-bottom: 16px;caret-color: rgb(36, 41, 46);color: rgb(36, 41, 46);font-family: system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;;line-height: 1.75em;"> <span style="font-size: 15px;">通过上述措施，可以有效地优化Elasticsearch集群在硬件层面的性能。</span> </section> </section>