Java学习者

作者：じ☆ve宝贝

<section data-role="outer" label="edit by 135editor"> <section style="background-position: 0% 0%;background-repeat: repeat;background-size: 23.5867%;background-attachment: scroll;font-size: 16px;color: rgb(62, 62, 62);"> <p style="text-align:center;"><img class="rich_pages wxw-img" data-galleryid="" data-imgfileid="100010495" data-ratio="0.39052287581699346" data-s="300,640" src="/upload/bda83b5dbe171be1259a9595182b73dc.jpg" data-type="jpeg" data-w="612" style="vertical-align: baseline;width: 100%;"></p> <section style="text-align: left;justify-content: flex-start;display: flex;flex-flow: row;margin-top: 10px;margin-bottom: 10px;"> <section style="display: inline-block;vertical-align: top;width: auto;flex: 100 100 0%;height: auto;align-self: flex-start;"> <section style="text-align: center;margin-top: 2px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> <section style="display: inline-block;vertical-align: middle;width: auto;align-self: center;flex: 0 0 auto;border-style: solid;border-width: 0px;border-left-color: rgb(109, 104, 247);padding-left: 9px;min-width: 5%;height: auto;line-height: 0.1;"> <section style="justify-content: flex-start;display: flex;flex-flow: row;"> <section style="display: inline-block;vertical-align: top;width: auto;align-self: stretch;flex: 0 0 auto;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);min-width: 5%;height: auto;padding: 4px;"> <section style="text-align: center;"> <section style="text-align: justify;line-height: 0.8;"> <p><strong><span style="color: rgb(255, 255, 255);text-align: center;">一、JWT简介</span></strong></p> </section> </section> </section> <section style="display: inline-block;vertical-align: top;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: stretch;"> <section style="text-align: center;height: 0px;"> <br> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: flex-end;"> <section style="transform-style: flat;transform: perspective(0px);-webkit-transform: perspective(0px);-moz-transform: perspective(0px);-o-transform: perspective(0px);"> <section style="transform: rotateX(180deg);-webkit-transform: rotateX(180deg);-moz-transform: rotateX(180deg);-o-transform: rotateX(180deg);"> <section style="display: inline-block;width: 8px;height: 8px;vertical-align: top;overflow: hidden;border-radius: 182px;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);"> <section style="text-align: justify;"> <p><br></p> </section> </section> </section> </section> </section> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;flex: 100 100 0%;height: auto;align-self: flex-end;padding-left: 8px;"> <section style="text-align: center;margin-bottom: 3px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> </section> <section> <p style="text-indent: 2em;"><em><span style="color: rgb(95, 156, 239);font-size: 14px;"><strong>JSON Web Token (JWT)</strong></span></em><span style="font-size: 14px;">是一种紧凑的、基于 JSON 的开放标准 (RFC 7519)，常用于不同主体（客户端和服务器）之间安全地传递信息。JWT 通常由三部分组成：</span></p> <ol class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p style="text-align:left;text-indent: 0em;"><strong><span style="font-size: 14px;text-indent: 2em;">Header（头部）：</span></strong><span style="font-size: 14px;text-indent: 2em;">定义令牌类型和加密算法，如{"alg": "HS256", "typ": "JWT"}</span></p></li> <li><p><strong><span style="font-size: 14px;">Payload（载荷）：</span></strong><span style="font-size: 14px;">包含声明信息（claims），如用户身份、权限等。</span></p></li> <li><p><strong><span style="font-size: 14px;">Signature（签名）：</span></strong><span style="font-size: 14px;">用来验证令牌的真实性和完整性。</span></p></li> </ol> <p><span style="font-size: 14px;">一个示例的 JWT：</span></p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js"><code><span class="code-snippet_outer">eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.</span></code><code><span class="code-snippet_outer">eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.</span></code><code><span class="code-snippet_outer">SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c</span></code></pre> </section> <p style="text-indent: 2em;"><span style="font-size: 14px;">JWT 被广泛用于身份认证，授权场景。服务端验证 JWT 时，可以通过解密签名来判断消息是否被篡改。</span></p> <p style="text-indent: 2em;"><br></p> </section> <section style="text-align: left;justify-content: flex-start;display: flex;flex-flow: row;margin-top: 10px;margin-bottom: 10px;"> <section style="display: inline-block;vertical-align: top;width: auto;flex: 100 100 0%;height: auto;align-self: flex-start;"> <section style="text-align: center;margin-top: 2px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> <section style="display: inline-block;vertical-align: middle;width: auto;align-self: center;flex: 0 0 auto;border-style: solid;border-width: 0px;border-left-color: rgb(109, 104, 247);padding-left: 9px;min-width: 5%;height: auto;line-height: 0.1;"> <section style="justify-content: flex-start;display: flex;flex-flow: row;"> <section style="display: inline-block;vertical-align: top;width: auto;align-self: stretch;flex: 0 0 auto;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);min-width: 5%;height: auto;padding: 4px;"> <section style="text-align: center;"> <section style="text-align: justify;line-height: 0.8;"> <p><strong><span style="color: rgb(255, 255, 255);text-align: center;">二、JWT自身攻击面</span></strong></p> </section> </section> </section> <section style="display: inline-block;vertical-align: top;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: stretch;"> <section style="text-align: center;height: 0px;"> <br> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: flex-end;"> <section style="transform-style: flat;transform: perspective(0px);-webkit-transform: perspective(0px);-moz-transform: perspective(0px);-o-transform: perspective(0px);"> <section style="transform: rotateX(180deg);-webkit-transform: rotateX(180deg);-moz-transform: rotateX(180deg);-o-transform: rotateX(180deg);"> <section style="display: inline-block;width: 8px;height: 8px;vertical-align: top;overflow: hidden;border-radius: 182px;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);"> <section style="text-align: justify;"> <p><br></p> </section> </section> </section> </section> </section> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;flex: 100 100 0%;height: auto;align-self: flex-end;padding-left: 8px;"> <section style="text-align: center;margin-bottom: 3px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> </section> <section style="font-size: 14px;"> <p style="text-align:left;"><span style="color: rgb(95, 156, 239);font-size: 15px;"><strong>2.1 算法混淆攻击（Algorithm Confusion Attack）/"None"算法攻击</strong></span></p> <p><br></p> <p><strong>技术要点：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>攻击者通过篡改 JWT 的 Header 部分，将签名算法从安全算法（如 HS256）更改为不安全的算法（如 none），从而伪造合法 Token。</p></li> <li><p>攻击者通过修改 JWT 的算法字段（如从 RSA 改为 HMAC），可以利用服务端验证机制的弱点来篡改令牌。例如，当服务端没有正确验证算法时，攻击者可能使用公钥重新签名令牌并将其发送回来，导致验证成功，即使令牌已经被篡改</p></li> </ul> <p><strong>示例：</strong></p> <p>原始 Header：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="json"><code><span class="code-snippet_outer">{<span class="code-snippet__attr">"alg"</span>: <span class="code-snippet__string">"HS256"</span>,<span class="code-snippet__attr">"typ"</span>: <span class="code-snippet__string">"JWT"</span>}</span></code></pre> </section> <p>攻击后被篡改为：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="json"><code><span class="code-snippet_outer">{<span class="code-snippet__attr">"alg"</span>: <span class="code-snippet__string">"none"</span>,<span class="code-snippet__attr">"typ"</span>: <span class="code-snippet__string">"JWT"</span>}</span></code></pre> </section> <p><br></p> <p>此时 JWT 将不再进行签名验证，攻击者可以伪造任意 Payload。例如：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="css"><code><span class="code-snippet_outer"><span class="code-snippet__selector-tag">eyJhbGciOiJub25lIiwidHlwIjoiSldUIn0</span><span class="code-snippet__selector-class">.eyJ1c2VySWQiOiIxMjMifQ</span>.</span></code></pre> </section> <p><strong>解决方案：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>在服务端严格校验 alg 字段，不允许使用 none 等不安全的算法。</p></li> <li><p>在生成和解析 JWT 时，明确指定并验证安全算法，如 HS256 或 HS512。</p></li> </ul> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="ini"><code><span class="code-snippet_outer"><span class="code-snippet__attr">payload</span> = jwt.decode(token, self.secret, algorithms=[<span class="code-snippet__string">"HS256"</span>, <span class="code-snippet__string">"HS512"</span>])</span></code><code><span class="code-snippet_outer"><span class="code-snippet__attr">userId</span> = payload[<span class="code-snippet__string">'userId'</span>]</span></code><code><span class="code-snippet_outer"><span class="code-snippet__attr">username</span> = self.db_lookup(userId, <span class="code-snippet__string">"username"</span>)</span></code></pre> </section> <p><br></p> <p><span style="color: rgb(95, 156, 239);font-size: 15px;"><strong>2.2 弱密钥导致Token 可伪造</strong></span></p> <p><br></p> <p><strong>技术要点：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>如果服务端使用弱密钥或者密钥管理不当，攻击者可以通过暴力破解或暴露的密钥生成伪造的 JWT，绕过验证。</p></li> <li><p>在对称加密（如 HMAC）中，JWT 的签名强度取决于密钥的复杂性。若使用弱密钥，攻击者可以通过暴力破解的方式获取密钥，生成伪造的 JWT。</p></li> </ul> <p><strong>示例：</strong>如果密钥过于简单，攻击者可以使用工具如 jwtcrack 破解签名：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="nginx"><code><span class="code-snippet_outer"><span class="code-snippet__attribute">jwtcrack</span> your_jwt_token</span></code></pre> </section> <p>破解后，攻击者可以使用这个密钥生成伪造的 JWT。</p> <p><br></p> <p><strong>解决方案：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>使用强加密算法，如 RS256（非对称加密），避免使用对称加密算法 HS256。</p></li> </ul> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="ini"><code><span class="code-snippet_outer"><span class="code-snippet__attr">payload</span> = jwt.decode(token, self.secret, algorithms=<span class="code-snippet__string">"RS256"</span>)</span></code></pre> </section> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>使用高强度的密钥，并妥善保管，避免泄露。</p><p><br></p></li> </ul> <p><span style="font-size: 15px;"><strong><span style="color: rgb(95, 156, 239);">2.3 Token 重放攻击</span></strong></span></p> <p><br></p> <p><strong>技术要点：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>攻击者可以拦截合法用户的 JWT 并在其有效期内重复使用，造成重放攻击。</p></li> <li><p>在验证 Token 签名之前，会计算 Token 的有效期，以确保 Token 尚未过期。通常是通过从 Token 中读取 exp （过期时间）声明并计算是否仍然有效来执行的。如果exp值设置得太大（或根本没有设置），Token 的有效时间就会太长，甚至可能永远不会过期。</p></li> </ul> <p><strong>示例：</strong>假设攻击者捕获了一个合法的 JWT，在其有效期内不断重放该请求以执行未授权操作。</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="http"><code><span class="code-snippet_outer"><span class="code-snippet__attribute">Authorization</span>: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...</span></code></pre> </section> <p><br></p> <p><strong>解决方案：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>设置较短的 exp 过期时间来限制 Token 的有效期。</p></li> </ul> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="makefile"><code><span class="code-snippet_outer">lifetime = datetime.datetime.now() + datetime.timedelta(minutes=5)</span></code><code><span class="code-snippet_outer">payload = {</span></code><code><span class="code-snippet_outer">    'username' : username,</span></code><code><span class="code-snippet_outer">    'admin' : 0,</span></code><code><span class="code-snippet_outer">    'exp' : lifetime</span></code><code><span class="code-snippet_outer">}</span></code><code><span class="code-snippet_outer">access_token = jwt.encode(payload, self.secret, algorithm=<span class="code-snippet__string">"HS256"</span>)</span></code></pre> </section> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>在 Token 生成过程中使用唯一标识符（如 nonce）防止重放攻击。</p></li> <li><p>在服务端维护 Token 使用历史，拒绝同一个 Token 被多次使用。</p><p><br></p></li> </ul> <p><span style="font-size: 15px;"><strong><span style="color: rgb(95, 156, 239);">2.4 Token 泄露与窃取风险</span></strong></span></p> <p><br></p> <p><strong>技术要点：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>如果 JWT 在不安全的传输渠道中传递，如使用 HTTP 而非 HTTPS，攻击者可以通过中间人攻击拦截并获取 JWT。</p></li> </ul> <p><br></p> <p><strong>解决方案：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>始终使用 HTTPS 传输 JWT，避免中间人攻击。</p></li> <li><p>避免在 URL 中传递 JWT，因为 URL 可能被日志记录。</p><p><br></p></li> </ul> <p><span style="color: rgb(95, 156, 239);font-size: 15px;"><strong>2.5 JWT Header参数注入伪造自签名</strong></span></p> <p style="text-indent: 2em;">JSON Web Signature (JWS) RFC 中定义的Header参数，最基本的JWT header是以下 JSON。</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="json"><code><span class="code-snippet_outer">{</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"typ"</span>: <span class="code-snippet__string">"JWT"</span>,</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"alg"</span>: <span class="code-snippet__string">"HS256"</span></span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <section style="text-indent: 2em;"> <span style="text-indent: 2em;letter-spacing: 0.034em;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, Arial, sans-serif;">其他在 RFC 中注册的Header参数包括：jwk、jku、kid等：</span> </section> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>jwk(JSON Web Key)：提供一个代表密钥的嵌入式JSON对象</p></li> <li><p>jku(JSON Web Key Set URL)：提供一个URL，服务器可以从这个URL获取一组包含正确密钥的密钥</p></li> <li><p>kid(Key ID)：提供一个ID，在有多个密钥可供选择的情况下服务器可以用它来识别正确的密钥，根据键的格式这可能有一个匹配的kid参数</p><p><br></p></li> </ul> <p style="text-indent: 2em;">这些用户可控制的Header参数每个都告诉服务端在验证签名时应该使用哪个密钥，如果服务端配置存在缺陷，通过这些参数的注入可伪造合法的自签名JWT。</p> <p><br></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>通过 jwk 参数注入自签名 JWT：</p></li> </ul> <p><strong>示例：</strong></p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="json"><code><span class="code-snippet_outer">{</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"kid"</span>: <span class="code-snippet__string">"ed2Nf8sb-sD6ng0-scs5390g-fFD8sfxG"</span>,</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"typ"</span>: <span class="code-snippet__string">"JWT"</span>,</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"alg"</span>: <span class="code-snippet__string">"RS256"</span>,</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"jwk"</span>: {</span></code><code><span class="code-snippet_outer">        <span class="code-snippet__attr">"kty"</span>: <span class="code-snippet__string">"RSA"</span>,</span></code><code><span class="code-snippet_outer">        <span class="code-snippet__attr">"e"</span>: <span class="code-snippet__string">"AQAB"</span>,</span></code><code><span class="code-snippet_outer">        <span class="code-snippet__attr">"kid"</span>: <span class="code-snippet__string">"ed2Nf8sb-sD6ng0-scs5390g-fFD8sfxG"</span>,</span></code><code><span class="code-snippet_outer">        <span class="code-snippet__attr">"n"</span>: <span class="code-snippet__string">"yy1wpYmffgXBxhAUJzHHocCuJolwDqql75ZWuCQ_cb33K2vh9m"</span></span></code><code><span class="code-snippet_outer">    }</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <p><strong>技术要点：</strong></p> <section style="text-indent: 2em;"> 理想情况下，服务端应仅使用有限的公钥白名单来验证 JWT 签名。但是，配置错误的服务端有时会使用jwk参数中嵌入的任何密钥。使用自己的RSA私钥对修改后的JWT进行签名，然后在jwk中嵌入匹配的公钥，服务端会使用jwk中嵌入的公钥验证 JWT 签名，导致伪造的自签名JWT通过验证。 </section> <p><br></p> <p><strong>解决方案：</strong></p> <p>服务端正确配置公钥白名单验证 JWT 签名</p> <p><br></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>通过 jku 参数注入自签名 JWT</p></li> </ul> <p><strong>示例：</strong></p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="php"><code><span class="code-snippet_outer">{<span class="code-snippet__string">"typ"</span>:<span class="code-snippet__string">"JWT"</span>,<span class="code-snippet__string">"alg"</span>:<span class="code-snippet__string">"RS256"</span>, <span class="code-snippet__string">"jku"</span>:<span class="code-snippet__string">"https://hacker.com/jwks.json"</span>, <span class="code-snippet__string">"kid"</span>:<span class="code-snippet__string">"id_of_jwks"</span>}.</span></code><code><span class="code-snippet_outer">{<span class="code-snippet__string">"login"</span>:<span class="code-snippet__string">"admin"</span>}.</span></code><code><span class="code-snippet_outer">[Signed with <span class="code-snippet__keyword">new</span> <span class="code-snippet__keyword">Private</span> key; <span class="code-snippet__keyword">Public</span> key exported]</span></code></pre> </section> <p><br></p> <p><strong>技术要点：</strong></p> <p style="text-indent: 2em;">与 jwk 不同的是，jku参数是指向 jwk 集合文件的 URL。攻击者将jku URL 替换为包含恶意公钥的 URL，再用配对的私钥对伪造的Token签名，服务端获取恶意公钥并验证伪造的Token为合法。</p> <p><br></p> <p><strong>解决方案：</strong></p> <p>服务端使用白名单验证jku参数值，仅允许指定来源的URL</p> <p><br></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>通过kid参数注入自签名JWT</p></li> </ul> <p><strong>示例：</strong></p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="json"><code><span class="code-snippet_outer">{</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"kid"</span>: <span class="code-snippet__string">"../../path/to/file"</span>,</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"typ"</span>: <span class="code-snippet__string">"JWT"</span>,</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"alg"</span>: <span class="code-snippet__string">"HS256"</span>,</span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">"k"</span>: <span class="code-snippet__string">"asGsADas3421-dfh9DGN-AFDFDbasfd8-anfjkvc"</span></span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <p><br></p> <p><strong>技术要点：</strong></p> <p style="text-indent: 2em;">服务端可能使用多个密钥对不同种类的数据签名，因此JWT的Header参数可能包含kid(Key ID)参数，让服务端在验证签名时确定使用哪个密钥，验证的密钥通常存储为一个JWK集合，在这种情况下服务端可能简单地查找与Token具有相同kid的JWK，然而JWS规范没有为kid定义具体的结构——它只是开发人员选择的任意字符串，例如:它们可能使用kid参数指向数据库中的特定条目，甚至是文件的名称，如果这个参数也存在目录遍历漏洞，则攻击者可能会强制服务端使用其文件系统中的任意文件作为验证密钥，例如 ../../../../../../../dev/null，由于这是一个空文件，读取它会返回一个空字符串。因此，使用空字符串对Token进行签名将得到能通过服务端验证的合法签名。</p> <p style="text-indent: 2em;"><br></p> <p><strong>解决方案：</strong></p> <section data-role="list"> <ul class="list-paddingleft-1" style="padding-left: 30px;list-style-position: outside;"> <li><p>服务端使用白名单验证kid参数值。</p></li> </ul> </section> <p><br></p> </section> <section style="text-align: left;justify-content: flex-start;display: flex;flex-flow: row;margin-top: 10px;margin-bottom: 10px;"> <section style="display: inline-block;vertical-align: top;width: auto;flex: 100 100 0%;height: auto;align-self: flex-start;"> <section style="text-align: center;margin-top: 2px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> <section style="display: inline-block;vertical-align: middle;width: auto;align-self: center;flex: 0 0 auto;border-style: solid;border-width: 0px;border-left-color: rgb(109, 104, 247);padding-left: 9px;min-width: 5%;height: auto;line-height: 0.1;"> <section style="justify-content: flex-start;display: flex;flex-flow: row;"> <section style="display: inline-block;vertical-align: top;width: auto;align-self: stretch;flex: 0 0 auto;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);min-width: 5%;height: auto;padding: 4px;"> <section style="text-align: center;"> <section style="text-align: justify;line-height: 0.8;"> <p><strong><span style="color: rgb(255, 255, 255);text-align: center;">三、JWT在业务场景的攻击面</span></strong></p> </section> </section> </section> <section style="display: inline-block;vertical-align: top;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: stretch;"> <section style="text-align: center;height: 0px;"> <br> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: flex-end;"> <section style="transform-style: flat;transform: perspective(0px);-webkit-transform: perspective(0px);-moz-transform: perspective(0px);-o-transform: perspective(0px);"> <section style="transform: rotateX(180deg);-webkit-transform: rotateX(180deg);-moz-transform: rotateX(180deg);-o-transform: rotateX(180deg);"> <section style="display: inline-block;width: 8px;height: 8px;vertical-align: top;overflow: hidden;border-radius: 182px;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);"> <section style="text-align: justify;"> <p><br></p> </section> </section> </section> </section> </section> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;flex: 100 100 0%;height: auto;align-self: flex-end;padding-left: 8px;"> <section style="text-align: center;margin-bottom: 3px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> </section> <section style="font-size: 14px;"> <p><span style="color: rgb(95, 156, 239);font-size: 15px;"><strong>3.1 敏感信息泄露</strong></span></p> <p><br></p> <p><strong>技术要点：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>JWT 的 Payload 是 Base64 编码，而非加密，因此内容可以被轻易解析。如果在 Payload 中包含敏感信息（如用户的身份、邮箱等），可能会造成信息泄露。</p></li> </ul> <p><strong>示例：</strong></p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="json"><code><span class="code-snippet_outer">{<span class="code-snippet__attr">"SSN"</span>: <span class="code-snippet__string">"123-45-6789"</span>,<span class="code-snippet__attr">"email"</span>: <span class="code-snippet__string">"user@example.com"</span>,<span class="code-snippet__attr">"role"</span>: <span class="code-snippet__string">"admin"</span>}</span></code></pre> </section> <p>攻击者可以轻易解码 Base64 部分，得到这些敏感数据。</p> <p><br></p> <p><strong>解决方案：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>避免在 JWT 的 Payload 中存储敏感信息，使用标识符（如 userId），而非明文信息。</p></li> </ul> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="ini"><code><span class="code-snippet_outer"><span class="code-snippet__attr">payload</span> = jwt.decode(token, self.secret, algorithms=<span class="code-snippet__string">"HS256"</span>)</span></code><code><span class="code-snippet_outer"><span class="code-snippet__attr">userId</span> = payload[<span class="code-snippet__string">'userId'</span>]</span></code><code><span class="code-snippet_outer"><span class="code-snippet__attr">password</span> = self.db_lookup(userId, <span class="code-snippet__string">"password"</span>)</span></code></pre> </section> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>如果必须包含敏感信息，应使用加密机制对 Payload 进行加密。</p></li> </ul> <p><br></p> <p><span style="color: rgb(95, 156, 239);font-size: 15px;"><strong>3.2 身份验证逻辑错误导致 JWT 可混用</strong></span></p> <p><br></p> <p><strong>技术要点：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>在某些场景中，如果不同身份类型的 JWT（如管理员和普通用户的 Token）没有严格区分，可能导致权限提升等问题。</p></li> </ul> <p><strong>示例：</strong> 某服务允许用户使用普通用户的 JWT 访问管理员接口，攻击者可以利用此漏洞提升权限：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="http"><code><span class="code-snippet_outer"><span class="code-snippet__keyword">POST</span> <span class="code-snippet__string">/admin/manage/add</span> HTTP/2</span></code><code><span class="code-snippet_outer"><span class="code-snippet__attribute">Authorization</span>: Bearer user_token_with_low_permissions</span></code></pre> </section> <p><strong>解决方案：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>在业务逻辑中增加对 Token 类型、权限的校验，确保不同身份的 Token 不能混用。</p></li> <li><p>在 JWT Payload 中明确用户的角色，并在服务端验证其权限。</p></li> <li><p>如果管理员和普通用户的 JWT 都是对称加密类型，则使用不同的密钥签名、验证。</p><p><br></p></li> </ul> <p><span style="color: rgb(95, 156, 239);font-size: 15px;"><strong>3.3 跨服务中继攻击</strong></span></p> <p><br></p> <p><strong>技术要点：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>在多服务场景中，如果未指定 audience 限制 Token 仅能访问指定的应用程序，可能导致A应用程序的Token能在B应用程序中合法使用，可能导致权限提升。</p></li> </ul> <p><br></p> <p><strong>示例：</strong> A服务允许用户使用B服务生成的 JWT 访问，攻击者可以利用此漏洞扩展访问权限：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="makefile"><code><span class="code-snippet_outer"><span class="code-snippet__section">Host:appA</span></span></code><code><span class="code-snippet_outer"><span class="code-snippet__section">Authorization: Bearer user_token_made_by_appB</span></span></code></pre> </section> <p><br></p> <p><strong>解决方案：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>多服务场景中，各服务单独验证 audience 防止其他服务的 Token 访问</p></li> </ul> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="ini"><code><span class="code-snippet_outer"><span class="code-snippet__attr">payload</span> = jwt.decode(token, self.secret, audience=[<span class="code-snippet__string">"appB"</span>], algorithms=<span class="code-snippet__string">"HS256"</span>)</span></code></pre> </section> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>如果各服务的JWT 都使用对称加密类型算法，则各服务使用不同的密钥签名、验证</p></li> </ul> <p><br></p> <p><span style="font-size: 15px;"><strong><span style="color: rgb(95, 156, 239);">3.4 注入与越权常规风险</span></strong></span></p> <p><br></p> <p><strong>技术要点：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>攻击者可能试图构造恶意的 JWT，以绕过权限检查或注入恶意数据，造成越权操作。</p></li> </ul> <p><br></p> <p><strong>示例：</strong> 攻击者构造如下 Payload，试图绕过权限检查：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="json"><code><span class="code-snippet_outer">{<span class="code-snippet__attr">"userId"</span>: <span class="code-snippet__string">"123"</span>,<span class="code-snippet__attr">"role"</span>: <span class="code-snippet__string">"admin"</span>}</span></code><code><span class="code-snippet_outer">{<span class="code-snippet__attr">"userId"</span>: <span class="code-snippet__string">"123"</span>,<span class="code-snippet__attr">"username"</span>: <span class="code-snippet__string">"admin' or 1=1#"</span>,<span class="code-snippet__attr">"password"</span>: <span class="code-snippet__string">"null"</span>}</span></code></pre> </section> <p>如果服务端没有严格验证 Token 的合法性，攻击者可能获得管理员权限。</p> <p><br></p> <p><strong>解决方案：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>严格验证 JWT 的签名、算法和格式，确保 Token 未被篡改。</p></li> <li><p>在每个业务接口中，正确处理 Token 权限校验，避免越权操作；对 JWT 中提交的数据进行过滤处理，防止恶意数据影响业务。</p><p><br></p></li> </ul> </section> <section style="text-align: left;justify-content: flex-start;display: flex;flex-flow: row;margin-top: 10px;margin-bottom: 10px;"> <section style="display: inline-block;vertical-align: top;width: auto;flex: 100 100 0%;height: auto;align-self: flex-start;"> <section style="text-align: center;margin-top: 2px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> <section style="display: inline-block;vertical-align: middle;width: auto;align-self: center;flex: 0 0 auto;border-style: solid;border-width: 0px;border-left-color: rgb(109, 104, 247);padding-left: 9px;min-width: 5%;height: auto;line-height: 0.1;"> <section style="justify-content: flex-start;display: flex;flex-flow: row;"> <section style="display: inline-block;vertical-align: top;width: auto;align-self: stretch;flex: 0 0 auto;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);min-width: 5%;height: auto;padding: 4px;"> <section style="text-align: center;"> <section style="text-align: justify;line-height: 0.8;"> <p><strong><span style="color: rgb(255, 255, 255);text-align: center;">四、JWT测试</span></strong></p> </section> </section> </section> <section style="display: inline-block;vertical-align: top;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: stretch;"> <section style="text-align: center;height: 0px;"> <br> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: flex-end;"> <section style="transform-style: flat;transform: perspective(0px);-webkit-transform: perspective(0px);-moz-transform: perspective(0px);-o-transform: perspective(0px);"> <section style="transform: rotateX(180deg);-webkit-transform: rotateX(180deg);-moz-transform: rotateX(180deg);-o-transform: rotateX(180deg);"> <section style="display: inline-block;width: 8px;height: 8px;vertical-align: top;overflow: hidden;border-radius: 182px;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);"> <section style="text-align: justify;"> <p><br></p> </section> </section> </section> </section> </section> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;flex: 100 100 0%;height: auto;align-self: flex-end;padding-left: 8px;"> <section style="text-align: center;margin-bottom: 3px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> </section> <section style="font-size: 14px;"> <p><span style="color: rgb(0, 0, 0);"><strong>针对JWT的完整测试流程</strong></span></p> <p><span style="color: rgb(0, 119, 255);">https://github.com/ticarpi/jwt_tool/wiki/Attack-Methodology </span></p> <p><br></p> <p><strong>开始：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>查找JWT</p><p><br></p></li> <li><p>查找测试接口</p><p><br></p></li> <li><p>重放请求检查JWT是否有效</p></li> </ul> <p><br></p> <p><strong>简单的检查：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>JWT是否必须的？</p><p><br></p></li> <li><p>JWT是否校验签名？</p><p><br></p></li> <li><p>JWT能否持续使用？</p><p><br></p></li> <li><p>JWT是否在客户端生成？</p><p><br></p></li> <li><p>接口是否先校验JWT再处理Payload？</p><p><br></p></li> <li><p>对称加密算法的JWT是否使用了弱密钥？</p></li> </ul> <p><br></p> <p><strong>测试已知漏洞：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>'none' Algorithm (CVE-2015-9235)</p><p><br></p></li> <li><p>RSA Key Confusion (CVE-2016-5431)</p><p><br></p></li> <li><p>JWKS Injection (CVE-2018-0114)</p><p><br></p></li> <li><p>null signature (CVE-2020-28042)</p></li> </ul> <p><br></p> <p><strong>测试其他漏洞：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>"kid" issues - reveal key & path traversal</p><p><br></p></li> <li><p>URL篡改攻击</p><p><br></p></li> <li><p>JWKS 欺骗</p></li> </ul> <p><br></p> <p><strong>额外检查：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>跨服务中继攻击/同服务身份验证逻辑错误</p><p><br></p></li> <li><p>JWT是否校验有效期exp</p></li> </ul> <p><br></p> <p><strong>更进一步：</strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>注入、越权等常规漏洞</p><p><br></p></li> <li><p>模糊测试</p><p><br></p></li> </ul> </section> <section style="text-align: left;justify-content: flex-start;display: flex;flex-flow: row;margin-top: 10px;margin-bottom: 10px;"> <section style="display: inline-block;vertical-align: top;width: auto;flex: 100 100 0%;height: auto;align-self: flex-start;"> <section style="text-align: center;margin-top: 2px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> <section style="display: inline-block;vertical-align: middle;width: auto;align-self: center;flex: 0 0 auto;border-style: solid;border-width: 0px;border-left-color: rgb(109, 104, 247);padding-left: 9px;min-width: 5%;height: auto;line-height: 0.1;"> <section style="justify-content: flex-start;display: flex;flex-flow: row;"> <section style="display: inline-block;vertical-align: top;width: auto;align-self: stretch;flex: 0 0 auto;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);min-width: 5%;height: auto;padding: 4px;"> <section style="text-align: center;"> <section style="text-align: justify;line-height: 0.8;"> <p><strong><span style="color: rgb(255, 255, 255);text-align: center;">五、相关工具</span></strong></p> </section> </section> </section> <section style="display: inline-block;vertical-align: top;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: stretch;"> <section style="text-align: center;height: 0px;"> <br> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;min-width: 5%;flex: 0 0 auto;height: auto;align-self: flex-end;"> <section style="transform-style: flat;transform: perspective(0px);-webkit-transform: perspective(0px);-moz-transform: perspective(0px);-o-transform: perspective(0px);"> <section style="transform: rotateX(180deg);-webkit-transform: rotateX(180deg);-moz-transform: rotateX(180deg);-o-transform: rotateX(180deg);"> <section style="display: inline-block;width: 8px;height: 8px;vertical-align: top;overflow: hidden;border-radius: 182px;background-image: linear-gradient(45deg, rgb(72, 213, 244) 0%, rgb(110, 94, 247) 100%);"> <section style="text-align: justify;"> <p><br></p> </section> </section> </section> </section> </section> </section> </section> <section style="display: inline-block;vertical-align: bottom;width: auto;flex: 100 100 0%;height: auto;align-self: flex-end;padding-left: 8px;"> <section style="text-align: center;margin-bottom: 3px;"> <section style="background-color: rgb(97, 163, 246);height: 1px;"> <svg viewbox="0 0 1 1" style="float: left;line-height: 0;width: 0px;vertical-align: top;"></svg> </section> </section> </section> </section> <section style="font-size: 14px;"> <p><strong><span style="color: rgb(97, 163, 246);">JWT.io</span></strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p><span style="color: rgb(0, 119, 255);">https://jwt.io/</span></p></li> <li><p>一个在线工具，可以解析和调试 JWT，帮助开发者查看 JWT 的内容和签名算法，识别常见问题。</p></li> </ul> <p><br></p> <p><strong><span style="color: rgb(95, 156, 239);">JWT Tool</span></strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p><span style="color: rgb(0, 119, 255);">https://github.com/ticarpi/jwt_tool</span></p></li> <li><p>用于分析、生成和攻击 JWT 的工具，支持算法混淆攻击等多种手段。</p></li> </ul> <p><br></p> <p><strong><span style="color: rgb(95, 156, 239);">jwtcrack</span></strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p><span style="color: rgb(0, 119, 255);">https://github.com/Sjord/jwtcrack</span></p></li> <li><p>字典枚举破解 HS256, HS384 或 HS512 加密算法的JWT</p></li> </ul> <p><br></p> <p><strong><span style="color: rgb(95, 156, 239);">CyberChef</span></strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p><span style="color: rgb(0, 119, 255);">https://gchq.github.io/CyberChef/</span></p></li> <li><p>用于JWT验证、解码、签名的在线工具</p></li> </ul> <p><br></p> <p><strong><span style="color: rgb(95, 156, 239);">JWS库</span></strong></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>提供生成和验证 JWT 的库，支持多种签名算法，可用于实现安全的 Token 处理逻辑。</p></li> </ul> <p><br></p> <p><span style="color: rgb(95, 156, 239);"><strong>Burp Suite 插件</strong></span></p> <ul class="list-paddingleft-1" style="list-style-position: outside;padding-left: 30px;"> <li><p>Burp Suite 插件市场</p></li> <li><p><span style="color: rgb(0, 0, 0);">sign-saboteur</span><span style="color: rgb(0, 119, 255);">[https://github.com/d0ge/sign-saboteur]</span><span style="letter-spacing: 0.034em;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, Arial, sans-serif;">：用于</span></p></li> </ul> <p style="letter-spacing: 0.578px;">编辑、签名、验证各种签名的 Web Token</p> <p><br></p> <p><br></p> </section> <p><br></p> <section class="mp_profile_iframe_wrp"> <mp-common-profile data-pluginname="mpprofile" data-id="Mzg2MDU5NjI0Mw==" data-headimg="https://mmbiz.qlogo.cn/mmbiz_png/U83RJGeEWqTI8bSZ1sQnibDyHl5ibOibViccRQJwtDINNWfYxfjrfwFBhtvxoq77HHwmbiaER9YLPlGqZaqrsuRgz2Q/0?wx_fmt=png" data-nickname="货拉拉安全应急响应中心" data-alias="lalasrc" data-signature="货拉拉安全应急响应中心（LLSRC）官方公众号" data-from="2" data-is_biz_ban="0"></mp-common-profile> </section> <p><br></p> </section> <p style="display: none;"><br></p> <section> <br> </section> <p><br></p> </section> <p style="display: none;"> <mp-style-type data-value="3"></mp-style-type></p>

Mybatis Plus 空字符串插入问题引起的思考

作者：微信小助手

别提桶跑路，MySQL 误删数据救命指南来了！

作者：微信小助手

<p data-mpa-powered-by="yiban.io"><img src="/upload/08a263ede13d33448d78713510cc9d81.png" class="rich_pages wxw-img" data-ratio="0.125" data-w="640" data-imgfileid="100076887"></p> <section data-tool="mdnice编辑器" data-website="https://www.mdnice.com" style="margin-bottom: 0px;padding-left: 10px;padding-right: 10px;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;font-family: Optima, "Microsoft YaHei", PingFangSC-regular, serif;font-size: 16px;color: rgb(0, 0, 0);line-height: 1.5em;word-spacing: 0em;letter-spacing: 0em;word-break: break-word;text-align: left;"> <h2 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="display: none;"></span><span style="font-size: 20px;color: rgb(0, 150, 136);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none solid;border-width: 1px 1px 1px 3px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">一、背景</span><span style="display: none;"></span></h2> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">某天，张三打算操作数据库，删除自己项目的无用数据，但是一不小心数据删多了。被误删的数据，如何恢复呢？本文将介绍相关方法，以及现成的一些工具。</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">例子：</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">有一个表</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #f92672;font-weight: bold;line-height: 26px;">create</span> <span style="color: #f92672;font-weight: bold;line-height: 26px;">table</span> person<br>(<br>  <span style="color: #f92672;font-weight: bold;line-height: 26px;">id</span>   <span style="color: #a6e22e;line-height: 26px;">bigint</span> primary <span style="color: #f92672;font-weight: bold;line-height: 26px;">key</span> auto_increment <span style="color: #f92672;font-weight: bold;line-height: 26px;">comment</span> <span style="color: #a6e22e;line-height: 26px;">'id'</span>,<br>  <span style="color: #f92672;font-weight: bold;line-height: 26px;">name</span> <span style="color: #a6e22e;line-height: 26px;">varchar</span>(<span style="line-height: 26px;">50</span>) <span style="color: #f92672;font-weight: bold;line-height: 26px;">comment</span> <span style="color: #a6e22e;line-height: 26px;">'名称'</span><br>) <span style="color: #f92672;font-weight: bold;line-height: 26px;">engine</span> = <span style="color: #f92672;font-weight: bold;line-height: 26px;">innodb</span>;<br></code></pre> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">原本是要执行这条SQL语句：</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #f92672;font-weight: bold;line-height: 26px;">delete</span> <span style="color: #f92672;font-weight: bold;line-height: 26px;">from</span> person <span style="color: #f92672;font-weight: bold;line-height: 26px;">where</span> <span style="color: #f92672;font-weight: bold;line-height: 26px;">id</span> > <span style="line-height: 26px;">500000</span>;<br></code></pre> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">不小心执行了这条SQL语句：</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #f92672;font-weight: bold;line-height: 26px;">delete</span> <span style="color: #f92672;font-weight: bold;line-height: 26px;">from</span> person;<br></code></pre> <h2 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="display: none;"></span><span style="font-size: 20px;color: rgb(0, 150, 136);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none solid;border-width: 1px 1px 1px 3px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">二、解决方案</span><span style="display: none;"></span></h2> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">处理这个问题的解决思路就是，基于binlog找回被删除的数据，将被删除的数据重新插入到数据库。</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">对于binlog文件来说，实际上保存的是对于数据库的正向操作。比如说，插入数据insert，binlog中保存的也是insert语句；删除数据delete，binlog中保存的也是delete语句。</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">因此，想要恢复被删除的数据，主要有两种方式：</p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100076885" data-ratio="0.43386243386243384" src="/upload/2b90a9edeaf25a7624d8cf7cbd987632.png" data-type="png" data-w="756" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">下面就针对上面的两种方式，进行详细的讲解</p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="display: none;"></span><span style="font-size: 18px;color: rgb(34, 34, 34);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none solid;border-width: 1px 1px 1px 2px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">1. 通用操作</span><span style="display: none;"></span></h3> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">首先介绍两种方式都需要使用到的一些通用的操作，主要用于设置binlog、找到binlog文件</p> <h5 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"></span><span style="color: rgb(34, 34, 34);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none dashed;border-width: 1px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;color: rgb(0, 0, 0);display: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span>1.1 确认binlog开启<span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;color: rgb(0, 0, 0);display: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span></span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"></span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span></h5> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><strong style="color: rgb(0, 150, 136);background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">1.1.1 查询开启状态</strong></p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">首先要保证binlog是开启的，不然数据肯定是没办法恢复回来的。</p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">在MySQL中，可以通过执行以下SQL查询来检查是否已经开启了binlog：</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #f92672;font-weight: bold;line-height: 26px;">SHOW</span> <span style="color: #f92672;font-weight: bold;line-height: 26px;">VARIABLES</span> <span style="color: #f92672;font-weight: bold;line-height: 26px;">LIKE</span> <span style="color: #a6e22e;line-height: 26px;">'log_bin'</span>;<br></code></pre> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">这个查询将返回一个结果集，其中包含名为<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: "Operator Mono", Consolas, Monaco, Menlo, monospace;word-break: break-all;">log_bin</code>的系统变量的值。如果<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: "Operator Mono", Consolas, Monaco, Menlo, monospace;word-break: break-all;">log_bin</code>的值为ON，则表示binlog已经开启；如果值为OFF，则表示binlog没有开启。</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;">mysql> <span style="color: #f92672;font-weight: bold;line-height: 26px;">SHOW</span> <span style="color: #f92672;font-weight: bold;line-height: 26px;">VARIABLES</span> <span style="color: #f92672;font-weight: bold;line-height: 26px;">LIKE</span> <span style="color: #a6e22e;line-height: 26px;">'log_bin'</span>;<br>+<span style="color: #75715e;line-height: 26px;">---------------+-------+</span><br>| Variable_name | Value |<br>+<span style="color: #75715e;line-height: 26px;">---------------+-------+</span><br>| log_bin       | ON    |<br>+<span style="color: #75715e;line-height: 26px;">---------------+-------+</span><br>1 row in <span style="color: #f92672;font-weight: bold;line-height: 26px;">set</span> (<span style="line-height: 26px;">0.01</span> sec)<br></code></pre> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><strong style="color: rgb(0, 150, 136);background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">1.1.2 开启binlog</strong></p> <p data-tool="mdnice编辑器" style="font-size: 15px;line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">如果发现没有开启，可以通过修改MySQL配置文件（通常是<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: "Operator Mono", Consolas, Monaco, Menlo, monospace;word-break: break-all;">my.cnf</code>或<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: "Operator Mono", Consolas, Monaco, Menlo, monospace;word-break: break-all;">my.ini</code>，Linux下MySQL的配置文件目录一般是<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: "Operator Mono", Consolas, Monaco, Menlo, monospace;word-break: break-all;">/etc/mysql</code>）中的[<code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;height: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: "Operator Mono", Consolas, Monaco, Menlo, monospace;word-break: break-all;">mysqld</code>]部分来开启binlog。如果在配置文件中找到了类似以下的设置，则表示binlog已经开启：</p> <pre data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;background: #272822;color: #ddd;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;">[mysqld]<br>log-bin=mysql-bin<br>server-id=1<br></code></pre> <ul data-tool="mdnice编辑器" style="list-style-type: square;margin-top: auto;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);font-size: 15px;line-height: 1.8em;letter-spacing: 0.06em;"> <p style="color: rgb(0, 0, 0);line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;">修改配置启用了binlog之后，需要重启MySQL服务才能使更改生效</p> </section></li> <li> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);font-size: 15px;line-height: 1.8em;letter-spacing: 0.06em;"> <p style="color: rgb(0, 0, 0);line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;height: auto;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: "Operator Mono", Consolas, Monaco, Menlo, monospace;word-break: break-all;">mysql-bin</code>表示binlog文件的前缀</p> </section></li> <li> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);font-size: 15px;line-height: 1.8em;letter-spacing: 0.06em;"> <p style="color: rgb(0, 0, 0);line-height: 1.8em;letter-spacing: 0.02em;text-align: justify;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><code style="color: rgb(0, 150, 136);font-size: 14px;line-height: 1.8em;letter-spacing: 0em;height: auto;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;margin-left: 2px;margin-right: 2px;padding: 2px 4px;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: "Operator Mono", Consolas, Monaco, Menlo, monospace;word-break: break-all;">server-id</code> 设置了MySQL服务器的唯一ID，必须设置ID，否则没办法开启binlog</p> </section></li> </ul> <h5 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;display: none;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"></span><span style="color: rgb(34, 34, 34);line-height: 1.8em;letter-spacing: 0em;padding-left: 10px;border-style: none none none dashed;border-width: 1px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 0, 0) rgb(0, 150, 136);border-radius: 0px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;box-shadow: none;display: block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;color: rgb(0, 0, 0);display: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span>1.2 binlog模式<span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 0px;box-shadow: none;color: rgb(0, 0, 0);display: none;flex-direction: unset;float: unset;height: auto;justify-content: unset;letter-spacing: 0px;line-height: 1.5em;overflow: unset;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">unset</span></span><span style="align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: no

Apache Paimon 在抖音集团多场景中的优化实践

作者：微信小助手

<section style="margin-bottom: 0px;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-imgfileid="100036198" data-ratio="0.2222222222222222" src="/upload/e6027109404f4c4053d14e8284b886e9.png" data-w="1080" style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: var(--articleFontsize);letter-spacing: 0.034em;"> <br> </section> <section style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);visibility: visible;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="color: rgb(178, 178, 178);font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 12px;letter-spacing: 1px;background-color: rgb(255, 255, 255);">本文将基于抖音集团内部两大业务的典型实时数仓场景，介绍Paimon在抖音集团内部的生产实践。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="246" data-backw="578" data-galleryid="" data-imgfileid="100036200" data-ratio="0.42592592592592593" data-s="300,640" src="/upload/4f674f3652c52532ab53344e3b5b4056.png" data-type="png" data-w="1080" style="width: 100%;height: auto;"> </section> <section style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);text-align: right;visibility: visible;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;">作者：李明、苏兴、文杰</strong></span> </section> <h1 style="-webkit-tap-highlight-color: transparent;margin-right: 8px;margin-left: 8px;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);text-align: right;visibility: visible;"></h1> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;">抖音集团大数据工程师</strong></span> <br> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <hr style="border-style: solid;border-width: 1px 0 0;border-color: rgba(0,0,0,0.1);-webkit-transform-origin: 0 0;-webkit-transform: scale(1, 0.5);transform-origin: 0 0;transform: scale(1, 0.5);"> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"></strong></span> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;">目前抖音集团内部主要使用 Lambda 架构进行实时数仓建设，其中实时处理链路主要采用 Flink + MQ 进行实现。</span> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;">在 Lambda 架构体系下，主要优势是数据新鲜度高，但采用两条处理链路也带来了其它问题：</span> <br> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">1. 维护成本高：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;">需要维护实时、离线两条不同技术栈的处理链路，开发和维护成本高；</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">2. 计算口径难对齐：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;">没有统一的 Table 抽象，Schema 难对齐；两条链路同时跑，计算语义难对齐；</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">3. OLAP 查询能力差：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;">消息队列只支持 APPEND 流，从流式数据转化为 Table 的成本开销高，需要不断处理 changelog，导致 OLAP 查询能力差；</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">4. 问题排查困难：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;">如果数据有问题，用户需要排查数据 Pipeline，但由于中间结果不可查，导致排查难度高；</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-size: 15px;letter-spacing: 1px;">5. 数据订正困难：</span></strong> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;">实时链路数据订正需要大量的人工介入，修改逻辑、双跑等，数据订正困难。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">随着流式计算引擎的不断完善，以 Flink 为代表的流式计算引擎提出了新的目标：</span> <strong><span style="font-size: 15px;letter-spacing: 1px;color: rgb(0, 128, 255);">为有限数据和无限数据提供一套统一的处理 API。</span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">流批一体的计算模式进一步简化了数仓生产体系，将计算链路收敛到了相同的技术栈，降低了开发和维护成本。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">同时随着数据湖技术的兴起，它能够支持高效的数据流 / 批读写、数据回溯以及数据更新，进一步解决了Lambda 架构体系下的其它问题。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">Apache Paimon是一项流式数据湖存储技术，基于Flink Table Store独立孵化出来的项目，<strong><span style="font-size: 15px;letter-spacing: 1px;color: rgb(0, 128, 255);">主要目标是解决流式场景中的常见问题，为用户提供高吞吐、低延迟的数据摄入、流式订阅和实时查询，</span></strong>支持主流的计算 / OLAP 引擎，尤其对 Flink 的支持最佳。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-size: 15px;letter-spacing: 1px;">因此在经过调研后，最终决定采用 Apache Paimon 作为数据湖底座，和业务进行新一代实时数仓建设。</span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"></span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">本文将基于抖音集团两个业务的典型实时数仓场景，介绍 Apache Paimon 在抖音集团内部的生产实践。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section data-mpa-template="t" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);visibility: visible;"> <section data-mpa-template="t" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"> <section data-mpa-category="模板" data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;"> <section style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;margin-left: 8px;margin-right: 8px;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;display: flex;align-items: flex-end;justify-content: space-between;border-bottom: 1px solid rgba(9, 9, 9, 0.55);visibility: visible;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;border-bottom: 6px solid rgb(0, 110, 255);visibility: visible;"> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;visibility: visible;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "Helvetica Neue", Helvetica, "Hiragino Sans GB", "Microsoft YaHei", Arial, sans-serif;visibility: visible;">01</span></p> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 26px;letter-spacing: 0.034em;">场景一：游戏视频指标上卷</span></p> </section> </section> </section> </section> </section> </section> <section style="-webkit-tap-highlight-color: transparent;margin: 8px 8px 0px;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"> <br> </section> <section style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 业务场景</span></strong> </section> <section style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">游戏-新游场景在公测宣发、测试上线首日、首发等相关节点，产品和运营需要根据游戏短视频的点赞、曝光、评论等实时指标在第一时间挖掘优质作者和发现潜力热点。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">游戏实时数仓团队当前通过接入短视频实时数仓团队的分钟粒度流并关联游戏相关维表，通过分钟粒度上卷到天粒度指标的方案来提供相关指标。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 原有方案</span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="162" data-backw="578" data-galleryid="" data-imgfileid="100036201" data-ratio="0.2796296296296296" data-s="300,640" src="/upload/8f073af6a895d40d61dbc62fa34c41d5.png" data-type="png" data-w="1080" style="width: 100%;height: auto;"> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"></span></strong> </section> <section style="-webkit-tap-highlight-color: transparent;margin-bottom: 0px;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><strong style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 方案痛点</span></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><strong style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">1. 由于短视频 topic 流量在 100w+/s 左右，即使 Lookup Join HitRate 平均在 90% 左右，但是全链路峰值仍有60w+/s 的流量打到维表存储，<strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;color: rgb(0, 128, 255);">给维表服务带来比较大的访问压力。</span></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="89" data-backw="578" data-height="534" data-imgfileid="100036203" data-ratio="0.1537037037037037" src="/upload/936ef7572e992859cf2a857fd6548e9f.png" data-type="png" data-w="1080" data-width="3482" style="width: 100%;height: auto;"> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);">Lookup Join HitRate</span></em></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <em><span style="color: rgb(143, 149, 158);font-size: 14px;"><br></span></em> </section> <section style="margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-height="538" data-imgfileid="100036202" data-ratio="0.15092592592592594" src="/upload/4cddf7ab18a154341ca83689ba12425a.png" data-type="png" data-w="1080" data-width="3560"> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);">Lookup Join Request Per Second </span></em></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);"><br></span></em></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">2. 由于上卷任务的 source 是 append 流，分钟粒度的指标会实时的变化，所以需要消费 source 后通过 MAX / LAST_VALUE 等聚合函数去构建 retract 流、处理乱序等问题，<strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;color: rgb(0, 128, 255);">开发效率低且增加额外的状态成本。</span></strong></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span data-lark-record-data="{"rootId":"Ms3tdWpIroo7Y0xkipzc5F7VnYd","text":{"initialAttributedTexts":{"text":{"0":"由于上卷任务的 source 是 append 流，分钟粒度的指标会实时的变化，所以需要消费 source 后通过 MAX / LAST_VALUE 等聚合函数去构建 retract 流、处理乱序等问题，开发效率低且增加额外的状态成本。"},"attribs":{"0":"*0+3a"}},"apool":{"numToAttrib":{"0":["author","6891832921678954498"]},"nextNum":1}},"type":"ordered","referenceRecordMap":{},"extra":{"channel":"saas","pasteRandomId":"ab0c606d-57dd-49d4-8c87-a553fcf2c0b1","mention_page_title":{},"external_mention_url":{}},"isKeepQuoteContainer":false,"isFromCode":false,"selection":[{"id":23,"type":"text","selection":{"start":0,"end":118},"recordId":"doxcnJmyG2OGJu9Rsdw4apWT2Pg"}],"payloadMap":{},"isCut":false}" data-lark-record-format="docx/text"></span> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">CREATE</span> <span class="code-snippet__keyword">VIEW</span> view_01 <span class="code-snippet__keyword">AS</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">SELECT</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f3) <span class="code-snippet__keyword">AS</span> f3,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> f5,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f6) <span class="code-snippet__keyword">AS</span> f6,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f7) <span class="code-snippet__keyword">AS</span> f7,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">LAST_VALUE</span>(f8, f5) <span class="code-snippet__keyword">AS</span> f8</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">FROM</span> source_table</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">GROUP</span> <span class="code-snippet__keyword">BY</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> f5;</span></code> <code><span class="code-snippet_outer"><br></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">INSERT</span> <span class="code-snippet__keyword">INTO</span> sink_table</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">SELECT</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">SUM</span>(f3) <span class="code-snippet__keyword">AS</span> f3,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">CAST</span>(f2 <span class="code-snippet__keyword">AS</span> <span class="code-snippet__built_in">BIGINT</span>) <span class="code-snippet__keyword">AS</span> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f4) <span class="code-snippet__keyword">AS</span> f4,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f5) <span class="code-snippet__keyword">AS</span> f5,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f6) <span class="code-snippet__keyword">AS</span> f6,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f7) <span class="code-snippet__keyword">AS</span> f7,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">LAST_VALUE</span>(f8, f5) <span class="code-snippet__keyword">AS</span> f8</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">FROM</span> view_01</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">GROUP</span> <span class="code-snippet__keyword">BY</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2;</span></code> </section></pre> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);"><br></span></em></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <span style="font-size: 12px;"><em><span style="font-size: 12px;color: rgb(143, 149, 158);"><br></span></em></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-size: 15px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ Paimon实践</span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong style="font-size: 15px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">使用 Paimon 作为游戏维表，在 Flink 中 Lookup Join 将打宽结果写入 Paimon 表中，Paimon 表基于 lookup changelog producer 产生完整的 changelog，下游消费 changelog 做上卷计算。在存储层基于 Paimon 的 Sequence Field 能力处理乱序。</span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="165" data-backw="578" data-galleryid="" data-imgfileid="100036204" data-ratio="0.2861111111111111" data-s="300,640" src="/upload/6a9f1b75d15431a6719a4bd6bde6f8be.png" data-type="png" data-w="1080" style="width: 100%;height: auto;"> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;"><br></span></strong> </section> <section style="margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;">1. 维表打宽</span></strong> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__comment">--维表模型DDL</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">create</span> <span class="code-snippet__keyword">table</span> dim_table01 (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`id`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f1`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f2`</span> <span class="code-snippet__built_in">BIGINT</span></span></code> <code><span class="code-snippet_outer"> PRIMART <span class="code-snippet__keyword">KEY</span> (f1) <span class="code-snippet__keyword">NOT</span> <span class="code-snippet__keyword">ENFORCED</span></span></code> <code><span class="code-snippet_outer">) <span class="code-snippet__keyword">WITH</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer'</span>=<span class="code-snippet__string">'lookup'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer.row-deduplicate'</span>=<span class="code-snippet__string">'true'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'sequence.field'</span>=<span class="code-snippet__string">'f2'</span>,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer">)</span></code> <code><span class="code-snippet_outer"> </span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">create</span> <span class="code-snippet__keyword">table</span> dim_table02 (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`id`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f1`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f2`</span> <span class="code-snippet__built_in">BIGINT</span></span></code> <code><span class="code-snippet_outer"> PRIMART <span class="code-snippet__keyword">KEY</span> (f1) <span class="code-snippet__keyword">NOT</span> <span class="code-snippet__keyword">ENFORCED</span></span></code> <code><span class="code-snippet_outer">) <span class="code-snippet__keyword">WITH</span>(</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer'</span>=<span class="code-snippet__string">'lookup'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer.row-deduplicate'</span>=<span class="code-snippet__string">'true'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'sequence.field'</span>=<span class="code-snippet__string">'f2'</span>,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer">)</span></code> <code><span class="code-snippet_outer"><br></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__comment">--分钟指标流关联维度</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">SELECT</span></span></code> <code><span class="code-snippet_outer"> AA.id,</span></code> <code><span class="code-snippet_outer"> BB.f1 <span class="code-snippet__keyword">as</span> bb_f1,</span></code> <code><span class="code-snippet_outer"> CC.f1 <span class="code-snippet__keyword">as</span> cc_f1</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">FROM</span> source_table AA</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">LEFT</span> <span class="code-snippet__keyword">JOIN</span></span></code> <code><span class="code-snippet_outer"> paimon.db_name.dim_table01 <span class="code-snippet__comment">/*+ OPTIONS('lookup.async'='true', 'lookup.async-thread-number'='8') */</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">FOR</span> SYSTEM_TIME <span class="code-snippet__keyword">AS</span> <span class="code-snippet__keyword">OF</span> proctime <span class="code-snippet__keyword">AS</span> BB</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">ON</span> AA.id = BB.id</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">LEFT</span> <span class="code-snippet__keyword">JOIN</span></span></code> <code><span class="code-snippet_outer"> paimon.db_name.dim_table02 <span class="code-snippet__comment">/*+ OPTIONS('lookup.async'='true', 'lookup.async-thread-number'='8') */</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">FOR</span> SYSTEM_TIME <span class="code-snippet__keyword">AS</span> <span class="code-snippet__keyword">OF</span> proctime <span class="code-snippet__keyword">AS</span> CC</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">ON</span> AA.id = CC.id;</span></code> </section></pre> </section> <section style="text-align: right;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(0, 0, 0);visibility: visible;font-size: 15px;letter-spacing: 1px;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(0, 0, 0);visibility: visible;font-size: 15px;letter-spacing: 1px;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;">2. 指标上卷</strong></span> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__comment">--分钟指标模型DDL</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">create</span> <span class="code-snippet__keyword">table</span> <span class="code-snippet__string">`db_name`</span>.<span class="code-snippet__string">`table_name`</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`id`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f1`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f2`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f3`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f4`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f5`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f6`</span> <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f7`</span> <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`f8`</span> <span class="code-snippet__keyword">map</span><<span class="code-snippet__keyword">STRING</span>, <span class="code-snippet__keyword">STRING</span>></span></code> <code><span class="code-snippet_outer"> PRIMARY <span class="code-snippet__keyword">KEY</span> (<span class="code-snippet__keyword">id</span>, f1, f2, f4, f5, <span class="code-snippet__built_in">date</span>, <span class="code-snippet__keyword">hour</span>) <span class="code-snippet__keyword">NOT</span> <span class="code-snippet__keyword">ENFORCED</span></span></code> <code><span class="code-snippet_outer">) PARTITIONED <span class="code-snippet__keyword">by</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`date`</span> <span class="code-snippet__keyword">STRING</span> <span class="code-snippet__keyword">comment</span> <span class="code-snippet__string">'日期'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">`hour`</span> <span class="code-snippet__keyword">STRING</span> <span class="code-snippet__keyword">comment</span> <span class="code-snippet__string">'小时'</span></span></code> <code><span class="code-snippet_outer">) <span class="code-snippet__keyword">WITH</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'changelog-producer'</span>=<span class="code-snippet__string">'lookup'</span>, </span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'partition.expiration-time'</span>=<span class="code-snippet__string">'30 d'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'partition.timestamp-pattern'</span>=<span class="code-snippet__string">'$date'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'partition.expiration-check-interval'</span>=<span class="code-snippet__string">'3h'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'sequence.field'</span>=<span class="code-snippet__string">'f8,f3'</span>,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer">);</span></code> <code><span class="code-snippet_outer"><br></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__comment">--分钟指标上卷</span></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">INSERT</span> <span class="code-snippet__keyword">INTO</span> sink_table</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">SELECT</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">SUM</span>(f3) <span class="code-snippet__keyword">AS</span> f3,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f4) <span class="code-snippet__keyword">AS</span> f4,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f5) <span class="code-snippet__keyword">AS</span> f5,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f6) <span class="code-snippet__keyword">AS</span> f6,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">MAX</span>(f7) <span class="code-snippet__keyword">AS</span> f7,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">LAST_VALUE</span>(f8, f7) <span class="code-snippet__keyword">AS</span> f8</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">FROM</span> paimon.db_name.table_name</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">GROUP</span> <span class="code-snippet__keyword">BY</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2;</span></code> </section></pre> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><br></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;letter-spacing: 1px;font-size: 12px;color: rgb(0, 0, 0);visibility: visible;"><strong style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"><strong style="font-size: 15px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 方案收益</span></strong></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">1. 流批一体模式开发：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">原有链路需要使用 MAX、LAST_VALUE 等函数来构造 retract 消息，以保证下游 SUM 计算结果正确，流与批的开发模式是分割的。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">基于 Paimon 存储数据并补齐 changelog，开发模式流与批是对齐的，获得流批一体的开发体验，提高了开发效率。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">2. 维表新鲜度更高：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">原有链路中为了减少维表服务压力，所以本地 Cache TTL设置为 50 min，数据新鲜度较低，在Paimon 维表中默认每 10s 会主动检查维表数据是否有更新，并主动更新本地缓存，数据新鲜度更高。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">3. 数据乱序问题：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">原链路中需要使用 LAST_VALUE 来处理数据乱序问题，增加了额外的状态成本。Paimon 合并数据时可以根据 sequence.field 来排序，从而在存储层解决数据乱序问题，不需要在 Flink 中维护状态。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <p style="text-align: center;"><img class="rich_pages wxw-img" data-galleryid="" data-imgfileid="100036220" data-ratio="0.42592592592592593" data-s="300,640" src="/upload/bf75fb0811044ae7354738d75dae2b99.png" data-type="png" data-w="1080" style=""></p> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section data-mpa-template="t" mpa-from-tpl="t" style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);visibility: visible;"> <section data-mpa-template="t" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"> <section data-mpa-category="模板" data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;"> <section style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;margin-left: 8px;margin-right: 8px;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;display: flex;align-items: flex-end;justify-content: space-between;border-bottom: 1px solid rgba(9, 9, 9, 0.55);visibility: visible;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;border-bottom: 6px solid rgb(0, 110, 255);visibility: visible;"> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;visibility: visible;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "Helvetica Neue", Helvetica, "Hiragino Sans GB", "Microsoft YaHei", Arial, sans-serif;visibility: visible;">02</span></p> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 26px;letter-spacing: 0.034em;">场景二：财经多流拼接</span></p> </section> </section> </section> </section> </section> </section> <section style="margin: 8px 8px 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"> <br> </section> <section style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;">/ 业务场景</span></strong> </section> <section style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);line-height: 1.6em;margin-left: 8px;margin-right: 8px;"> <strong style="-webkit-tap-highlight-color: transparent;outline: 0px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 20px;"><br></span></strong> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">由于前端业务过程是由于多个功能模块通过接口拼接而成的，历史链路无法打通，关键节点指标缺失，<strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">DA同学需要投入大量时间找数据拼凑数据，用数成本高、数据精度低、无法实时看到增长策略表现。</span></strong></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">想要获得完整的用户交易过程的明细数据，财经侧在发起支付交易的时候创建了 trace_id，然后从支付开始透传到用户支付结束的所有流程中，利用 trace_id 构建用户交易行为宽表；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">宽表打宽的过程中，会去组合前后端埋点、会员、标签、策略、交易、营销等多个主题域的数据，降低业务同学用数、找数成本。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);">/ 原有业务方案</span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><br></span></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="272" data-backw="578" data-galleryid="" data-imgfileid="100036205" data-ratio="0.4703296703296703" data-s="300,640" src="/upload/ef317cebe62376e63518c5288e195203.png" data-type="png" data-w="910" style="width: 100%;height: auto;"> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">insert</span> <span class="code-snippet__keyword">into</span> sink</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">select</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">last_value</span>(f1) <span class="code-snippet__keyword">as</span> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">last_value</span>(f2) <span class="code-snippet__keyword">as</span> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">last_value</span>(f3) <span class="code-snippet__keyword">as</span> f3,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">last_value</span>(f4) <span class="code-snippet__keyword">as</span> f4,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">from</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">select</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f3,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f4,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">from</span> table1</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">union</span> <span class="code-snippet__keyword">all</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">select</span> <span class="code-snippet__keyword">id</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f2,</span></code> <code><span class="code-snippet_outer"> f3,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">from</span> table2</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">union</span> <span class="code-snippet__keyword">all</span></span></code> <code><span class="code-snippet_outer"> ......</span></code> <code><span class="code-snippet_outer"> )</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">group</span> <span class="code-snippet__keyword">by</span> <span class="code-snippet__keyword">id</span>;</span></code> </section></pre> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);">/ 方案痛点</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">1. 资源开销和运维成本高：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">当前基于 MQ 和 Flink 做打宽任务的多流 Join，状态大小超过了 10TB，计算资源 1600+ CU。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">大状态会导致资源开销变高、任务吞吐抖动、故障恢复时间变长等问题。在当前场景下，宽表任务的异常抖动会带来下游超 10 min 的断流感知，运维成本较高，亟需优化。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="213" data-backw="562" data-height="1170" data-imgfileid="100036206" data-ratio="0.37962962962962965" src="/upload/2c5d33b6c3354307b2f63e19824ec00e.png" data-type="png" data-w="1080" data-width="3080" style="width: 100%;height: auto;"> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="color: rgb(0, 128, 255);"><strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">2. 数据乱序问题：</span></strong></span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">由于打宽任务的超大状态，因此在 Flink 任务中状态的 TTL 配置相对较小（小时级）。在状态过期后，乱序数据会导致拼接结果不正确问题，产生额外的运维和排查成本</span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);">/ Paimon实践</span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><br></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">结合 Paimon 的 Partial Update 能力，对财经用户行为打宽任务进行了改写，数据直接写入到 Paimon 表，原链路中的聚合算子得以消除，任务状态大幅下降。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">同时基于 Paimon 的 Sequence Group 机制，为每个流定义了相应字段的顺序，避免因为乱序出现的数据不一致问题。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: center;margin-left: 8px;margin-right: 8px;"> <img class="rich_pages wxw-img" data-backh="249" data-backw="578" data-galleryid="" data-imgfileid="100036207" data-ratio="0.43037974683544306" data-s="300,640" src="/upload/9a58efb438c2d37a0fd1b28eb8bee6b3.png" data-type="png" data-w="948" style="width: 100%;height: auto;"> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"></span> </section> <section style="margin-left: 8px;margin-right: 8px;"> <br> </section> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="sql"> <section style="margin-left: 8px;margin-right: 8px;"> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">CREATE</span> <span class="code-snippet__keyword">TABLE</span> t (</span></code> <code><span class="code-snippet_outer"> trace_id <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> f1 <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> f2 <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> g_1 <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> f3 <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> f4 <span class="code-snippet__keyword">STRING</span>,</span></code> <code><span class="code-snippet_outer"> g_2 <span class="code-snippet__built_in">BIGINT</span>,</span></code> <code><span class="code-snippet_outer"> PRIMARY <span class="code-snippet__keyword">KEY</span> (trace_id) <span class="code-snippet__keyword">NOT</span> <span class="code-snippet__keyword">ENFORCED</span></span></code> <code><span class="code-snippet_outer">) <span class="code-snippet__keyword">WITH</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'merge-engine'</span>=<span class="code-snippet__string">'partial-update'</span>,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'fields.g_1.sequence-group'</span>=<span class="code-snippet__string">'f1,f2'</span>, <span class="code-snippet__comment">-- f1,f2字段根据 g_1 排序</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__string">'fields.g_2.sequence-group'</span>=<span class="code-snippet__string">'f3,f4'</span> <span class="code-snippet__comment">-- f3,f4字段根据 g_2 排序</span></span></code> <code><span class="code-snippet_outer">);</span></code> <code><span class="code-snippet_outer"><br></span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">insert</span> <span class="code-snippet__keyword">into</span> t</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">select</span> trace_id,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> g_1,</span></code> <code><span class="code-snippet_outer"> f3,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> g_2,</span></code> <code><span class="code-snippet_outer"> ...</span></code> <code><span class="code-snippet_outer"><span class="code-snippet__keyword">from</span> (</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">select</span> trace_id,</span></code> <code><span class="code-snippet_outer"> f1,</span></code> <code><span class="code-snippet_outer"> f2,</span></code> <code><span class="code-snippet_outer"> g_1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f3,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f4,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__built_in">BIGINT</span>) <span class="code-snippet__keyword">as</span> g_2,</span></code> <code><span class="code-snippet_outer"> xxx</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">from</span> table1</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">union</span> <span class="code-snippet__keyword">all</span></span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">select</span> trace_id,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f1,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__keyword">STRING</span>) <span class="code-snippet__keyword">as</span> f2,</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">cast</span>(<span class="code-snippet__literal">null</span> <span class="code-snippet__keyword">as</span> <span class="code-snippet__built_in">BIGINT</span>) <span class="code-snippet__keyword">as</span> g_1,</span></code> <code><span class="code-snippet_outer"> f3,</span></code> <code><span class="code-snippet_outer"> f4,</span></code> <code><span class="code-snippet_outer"> g_2,</span></code> <code><span class="code-snippet_outer"> xxx</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">from</span> table2</span></code> <code><span class="code-snippet_outer"> <span class="code-snippet__keyword">union</span> <span class="code-snippet__keyword">all</span></span></code> <code><span class="code-snippet_outer"> ......</span></code> <code><span class="code-snippet_outer"> )</span></code> </section></pre> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);">/ 方案收益</span><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"></span></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><span style="font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 20px;font-weight: 700;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"><br></span></span></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">1. 计算资源下降：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">计算资源优化 50%+ (1600 CU 缩减到 800 CU)，收益主要来源于状态管理成本下降；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">2. 状态优化：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">消除聚合状态后，作业状态由 10TB 缩减到小于 20GB；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">3. 开发和运维成本下降：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">中间数据可查可复用，同时指标增减可以通过 DDL 直接操作 Paimon 表；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;text-align: justify;font-size: 15px;letter-spacing: 1px;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">4. 数据乱序问题解决成本低：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;text-align: justify;font-size: 15px;letter-spacing: 1px;">基于 Paimon 的 Sequence Group 机制可以对多流数据按序进行合并，处理更长时间范围内的乱序问题，并且不额外新增状态成本，较之于原链路方案，数据质量提升6%</span> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: var(--articleFontsize);letter-spacing: 0.034em;text-align: justify;">。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <p style="text-align: center;"><img class="rich_pages wxw-img" data-galleryid="" data-imgfileid="100036221" data-ratio="0.44722222222222224" data-s="300,640" src="/upload/2630e23b908a6cc008b6194ae4c135f4.png" data-type="png" data-w="1080" style=""></p> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: var(--articleFontsize);letter-spacing: 0.034em;text-align: justify;"><br></span> </section> <section data-mpa-template="t" mpa-from-tpl="t" style="margin-bottom: 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);visibility: visible;"> <section data-mpa-template="t" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;visibility: visible;"> <section data-mpa-category="模板" data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;"> <section style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;visibility: visible;margin-left: 8px;margin-right: 8px;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 578.148px;display: flex;align-items: flex-end;justify-content: space-between;border-bottom: 1px solid rgba(9, 9, 9, 0.55);visibility: visible;"> <section data-mid="" mpa-from-tpl="t" style="-webkit-tap-highlight-color: transparent;outline: 0px;border-bottom: 6px solid rgb(0, 110, 255);visibility: visible;"> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;visibility: visible;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "Helvetica Neue", Helvetica, "Hiragino Sans GB", "Microsoft YaHei", Arial, sans-serif;visibility: visible;">03</span></p> <p data-mid="" style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 55px;color: rgb(0, 0, 0);font-weight: bold;word-break: break-word;line-height: 47px;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;font-size: 26px;letter-spacing: 0.034em;">未来展望</span></p> </section> </section> </section> </section> </section> </section> <section style="margin: 8px 8px 0px;-webkit-tap-highlight-color: transparent;outline: 0px;font-family: "PingFang SC", system-ui, -apple-system, "system-ui", "Helvetica Neue", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">我们期望未来以 Flink 为核心，以数据湖为底座，为用户提供全链路数据生产和管理的实时数仓解决方案，进一步简化用户的开发和使用成本。</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">我们也将继续针对实际业务场景进行 Apache Paimon 优化，包括：</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">1. 千列大宽表合并性能优化：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">LSM Tree 架构使得 Apache Paimon 有很高的点查与合并性能，但在超大列数的业务场景中性能下降较多，内部将针对这一场景进行优化；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <br> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">2. 维表性能优化：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">Apache Paimon 的本地维表可以极大的减少传统外部 KV 存储的请求数量，但在大流量场景中，我们注意到本地维表刷新是同步的，同时没有按照 bucket 进行 shuffle，导致维表变化较快时，吞吐有明显尖刺，我们将结合 Flink 继续优化维表的访问性能；</span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;"><br></span> </section> <section style="text-align: left;margin-left: 8px;margin-right: 8px;"> <strong><span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;color: rgb(0, 128, 255);">3. Merge Engine 扩展：</span></strong> <span style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 15px;letter-spacing: 1px;text-align: justify;">部分业务场景中，业务需要自定义的 Merge Engine 来实现更加复杂的合并策略，因此我们将扩展 Merge Engine，使其支持业务进行扩展以应对更加复杂的业务场景。</span> </section>

Spring Boot性能提升的核武器，速度提升500%！

作者：微信小助手

<p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">虚拟线程是 Java 21 引入的一个新特性，用于简化并发编程。它与传统的操作系统线程相比，具有显著的优势：</p> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">轻量级</span>：虚拟线程由 JVM 管理，而非操作系统，因此它们的内存占用和创建成本远低于传统线程。理论上，你可以轻松创建数十万甚至更多的虚拟线程。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">高并发性</span>：虚拟线程能处理更高并发的场景，特别是 I/O 密集型的应用，适合开发高并发、响应式的应用程序。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">自动管理</span>：无需手动管理线程池，JVM 会根据负载自动调整虚拟线程的调度，简化了并发编程的复杂性。 </section></li> </ol> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">虚拟线程的基础用法</span></h2> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">创建虚拟线程非常简单。你可以像创建传统线程一样启动虚拟线程，但它的创建与启动更加轻量：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer">Thread virtualThread = Thread.ofVirtual().start(() -> {</span></code><code><span class="code-snippet_outer"> System.<span class="code-snippet__keyword">out</span>.println(<span class="code-snippet__string">"虚拟线程正在运行"</span>);</span></code><code><span class="code-snippet_outer">});</span></code><code><span class="code-snippet_outer">System.<span class="code-snippet__keyword">out</span>.println(<span class="code-snippet__string">"主线程正在运行"</span>);</span></code></pre> </section> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">虚拟线程的延迟启动：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer">Thread virtualThread = Thread.ofVirtual()</span></code><code><span class="code-snippet_outer"> .name(<span class="code-snippet__string">"虚拟线程"</span>)</span></code><code><span class="code-snippet_outer"> .unstarted(() -> System.<span class="code-snippet__keyword">out</span>.println(<span class="code-snippet__string">"虚拟线程运行中"</span>));</span></code><code><span class="code-snippet_outer"><br></span></code><code><span class="code-snippet_outer">virtualThread.start();</span></code><code><span class="code-snippet_outer">virtualThread.<span class="code-snippet__keyword">join</span>(); <span class="code-snippet__comment">// 等待虚拟线程完成</span></span></code></pre> </section> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">在Spring Boot中使用虚拟线程</span></h2> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">在 Spring Boot 项目中使用虚拟线程需要一些简单的配置：</p> <section style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;word-spacing: 0.8px;background-color: rgb(255, 255, 255);text-align: center;margin-bottom: 16px;"> <img class="rich_pages wxw-img" src="https://studyjava.cn/files/articleCover/zsljava11733930693738884.png" data-cropx1="0" data-cropx2="941.6760563380282" data-cropy1="0" data-cropy2="445.43309859154925" data-galleryid="" data-imgfileid="100045784" data-ratio="0.4729011689691817" data-s="300,640" src="/upload/f04f513fabb195d2446283737613f997.png" data-type="jpeg" data-w="941" style="-webkit-tap-highlight-color: transparent;outline: 0px;width: 556px !important;visibility: visible !important;"> </section> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;margin-bottom: 16px;"> <span style="font-size: 16px;"><span style="font-size: 16px;-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">确保 Java 版本为 21 或以上</span>。</span> </section> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 在  <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">pom.xml</code> 中启用  <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">--enable-preview</code>，以便支持虚拟线程特性。 </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="xml"><code><span class="code-snippet_outer"><span class="code-snippet__tag"><<span class="code-snippet__name">plugin</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"><<span class="code-snippet__name">groupId</span>></span>org.apache.maven.plugins<span class="code-snippet__tag"></<span class="code-snippet__name">groupId</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"><<span class="code-snippet__name">artifactId</span>></span>maven-compiler-plugin<span class="code-snippet__tag"></<span class="code-snippet__name">artifactId</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"><<span class="code-snippet__name">configuration</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"><<span class="code-snippet__name">source</span>></span>21<span class="code-snippet__tag"></<span class="code-snippet__name">source</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"><<span class="code-snippet__name">target</span>></span>21<span class="code-snippet__tag"></<span class="code-snippet__name">target</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"><<span class="code-snippet__name">compilerArgs</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"><<span class="code-snippet__name">arg</span>></span>--enable-preview<span class="code-snippet__tag"></<span class="code-snippet__name">arg</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"></<span class="code-snippet__name">compilerArgs</span>></span></span></code><code><span class="code-snippet_outer"> <span class="code-snippet__tag"></<span class="code-snippet__name">configuration</span>></span></span></code><code><span class="code-snippet_outer"><span class="code-snippet__tag"></<span class="code-snippet__name">plugin</span>></span></span></code></pre> </section> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 在  <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">application.properties</code> 中启用性能监控工具： </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="makefile"><code><span class="code-snippet_outer">management.endpoints.web.exposure.<span class="code-snippet__keyword">include</span>=health,info,metrics</span></code></pre> </section> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 在 Spring Boot 中为 Tomcat 配置虚拟线程执行器： </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="css"><code><span class="code-snippet_outer">@<span class="code-snippet__keyword">Bean</span></span></code><code><span class="code-snippet_outer">public TomcatProtocolHandlerCustomizer<?> protocolHandlerVirtualThreadExecutorCustomizer() {</span></code><code><span class="code-snippet_outer"> <span class="code-snippet__selector-tag">return</span> <span class="code-snippet__selector-tag">protocolHandler</span> <span class="code-snippet__selector-tag">-</span>> <span class="code-snippet__selector-tag">protocolHandler</span><span class="code-snippet__selector-class">.setExecutor</span>(<span class="code-snippet__selector-tag">Executors</span><span class="code-snippet__selector-class">.newVirtualThreadPerTaskExecutor</span>());</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">实验：传统线程 vs 虚拟线程</span></h2> <h3 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 30px;margin-bottom: 15px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(34, 34, 34);line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1. 创建100,000个线程并执行</span></h3> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">传统线程</span>：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer"><span class="code-snippet__keyword">for</span> (<span class="code-snippet__keyword">int</span> i = <span class="code-snippet__number">0</span>; i < <span class="code-snippet__number">100</span>_000; i++) {</span></code><code><span class="code-snippet_outer"> Thread thread = <span class="code-snippet__keyword">new</span> Thread(() -> System.<span class="code-snippet__keyword">out</span>.println(i));</span></code><code><span class="code-snippet_outer"> thread.start();</span></code><code><span class="code-snippet_outer"> thread.<span class="code-snippet__keyword">join</span>();</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">执行耗时约 <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">18.6 秒</span>。</p> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">虚拟线程</span>：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer"><span class="code-snippet__keyword">for</span> (<span class="code-snippet__keyword">int</span> i = <span class="code-snippet__number">0</span>; i < <span class="code-snippet__number">100</span>_000; i++) {</span></code><code><span class="code-snippet_outer"> Thread thread = Thread.ofVirtual().unstarted(() -> System.<span class="code-snippet__keyword">out</span>.println(i));</span></code><code><span class="code-snippet_outer"> thread.start();</span></code><code><span class="code-snippet_outer"> thread.<span class="code-snippet__keyword">join</span>();</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">执行耗时仅 <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">3.7 秒</span>，性能提升了近 <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">500%</span> 。</p> <h3 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 30px;margin-bottom: 15px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(34, 34, 34);line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2. HTTP 请求性能对比</span></h3> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">在高并发场景下，虚拟线程的优势尤为明显。我们对比了传统线程与虚拟线程在处理 HTTP 请求时的表现。</p> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">配置 HTTP 线程执行器</span>：</p> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="css"><code><span class="code-snippet_outer">@<span class="code-snippet__keyword">Bean</span></span></code><code><span class="code-snippet_outer">public TomcatProtocolHandlerCustomizer<?> protocolHandlerVirtualThreadExecutorCustomizer() {</span></code><code><span class="code-snippet_outer"> <span class="code-snippet__selector-tag">return</span> <span class="code-snippet__selector-tag">protocolHandler</span> <span class="code-snippet__selector-tag">-</span>> <span class="code-snippet__selector-tag">protocolHandler</span><span class="code-snippet__selector-class">.setExecutor</span>(<span class="code-snippet__selector-tag">Executors</span><span class="code-snippet__selector-class">.newVirtualThreadPerTaskExecutor</span>());</span></code><code><span class="code-snippet_outer">}</span></code></pre> </section> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;"><span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">请求测试</span>：发送 1600 个 HTTP 请求，400 并发。</p> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">传统线程</span>： </section></li> <ul class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;list-style-type: disc;"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 请求耗时： <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">9.659 秒</span> </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 每秒请求数： <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">165.65</span> </section></li> </ul> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">虚拟线程</span>： </section></li> <ul class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;list-style-type: disc;"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 请求耗时： <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">7.912 秒</span> </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 每秒请求数： <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">202.22</span> </section></li> </ul> </ol> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">虚拟线程的吞吐量大幅提升，响应时间显著缩短。</p> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">Java性能提升的其他技巧</span></h2> <p data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;margin-bottom: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;">除了虚拟线程，Java 还有一些其他的性能提升技巧，尤其适用于 Spring Boot 高并发场景：</p> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">使用并行流</span>：对于 CPU 密集型任务，可以使用并行流（ <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">parallelStream()</code>）来利用多核 CPU，提高处理速度。 </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="typescript"><code><span class="code-snippet_outer">List<Integer> numbers = Arrays.asList(<span class="code-snippet__number">1</span>, <span class="code-snippet__number">2</span>, <span class="code-snippet__number">3</span>, <span class="code-snippet__number">4</span>, <span class="code-snippet__number">5</span>);</span></code><code><span class="code-snippet_outer">numbers.parallelStream().forEach(<span class="code-snippet__built_in">number</span> -> {</span></code><code><span class="code-snippet_outer"> System.out.println(<span class="code-snippet__built_in">number</span> * <span class="code-snippet__number">2</span>);</span></code><code><span class="code-snippet_outer">});</span></code></pre> </section> <ol start="2" data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">异步编程与CompletableFuture</span>：对于 I/O 密集型任务，可以使用  <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">CompletableFuture</code> 进行异步处理，减少线程阻塞，提高响应性能。 </section></li> </ol> <section class="code-snippet__fix code-snippet__js"> <ul class="code-snippet__line-index code-snippet__js"> </ul> <pre class="code-snippet__js" data-lang="cs"><code><span class="code-snippet_outer">CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {</span></code><code><span class="code-snippet_outer"> <span class="code-snippet__comment">// 异步执行任务</span></span></code><code><span class="code-snippet_outer"> System.<span class="code-snippet__keyword">out</span>.println(<span class="code-snippet__string">"异步任务完成"</span>);</span></code><code><span class="code-snippet_outer">});</span></code><code><span class="code-snippet_outer">future.<span class="code-snippet__keyword">join</span>(); <span class="code-snippet__comment">// 等待任务完成</span></span></code></pre> </section> <ol start="3" data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">优化数据库查询</span>：减少数据库查询的次数，使用缓存（如 Redis）来存储频繁查询的数据，减少不必要的 I/O 操作。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">内存管理优化</span>：通过使用对象池（如  <code style="-webkit-tap-highlight-color: transparent;margin-right: 2px;margin-left: 2px;padding: 2px 4px;outline: 0px;font-size: 14px;line-height: 1.8em;letter-spacing: 0em;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgba(27, 31, 35, 0.05);width: auto;border-style: none;border-width: 3px;border-color: rgb(0, 0, 0) rgba(0, 0, 0, 0.4) rgba(0, 0, 0, 0.4);border-radius: 4px;font-family: Consolas, Monaco, Menlo, monospace;word-break: break-all;color: rgb(255, 93, 108);">Apache Commons Pool</code>）来管理资源，减少频繁的对象创建和销毁，提高内存使用效率。 </section></li> </ol> <h2 data-tool="mdnice编辑器" style="-webkit-tap-highlight-color: transparent;margin-top: 20px;margin-right: 10px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);"><span style="-webkit-tap-highlight-color: transparent;padding-bottom: 10px;outline: 0px;font-size: 18px;color: rgb(234, 84, 41);line-height: 1.5em;letter-spacing: 0.5444px;font-weight: bold;display: block;border-bottom: 2px solid rgb(234, 84, 41);visibility: visible;">小结</span></h2> <ol data-tool="mdnice编辑器" class="list-paddingleft-1" style="-webkit-tap-highlight-color: transparent;margin-top: 8px;margin-bottom: 8px;padding-left: 25px;outline: 0px;font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;letter-spacing: 0.8px;text-align: left;word-spacing: 0.8px;background-color: rgb(255, 255, 255);color: rgb(248, 57, 41);"> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="-webkit-tap-highlight-color: transparent;outline: 0px;color: rgb(248, 57, 41);font-weight: 700;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;height: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;">虚拟线程</span> 是 Java 并发编程的革新，它简化了线程管理，提升了高并发场景下的性能。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 使用虚拟线程，你可以轻松创建数十万甚至更多线程，而不会影响应用的性能。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 在 Spring Boot 中配置虚拟线程非常简单，只需几行代码即可启用虚拟线程，带来显著的性能提升。 </section></li> <li style="-webkit-tap-highlight-color: transparent;outline: 0px;"> <section style="-webkit-tap-highlight-color: transparent;margin-top: 5px;margin-bottom: 5px;outline: 0px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> 除了虚拟线程，其他优化技巧（如并行流、异步编程、数据库查询优化等）也能有效提升 Java 应用的性能。 </section></li> </ol> <section style="-webkit-tap-highlight-color: transparent;margin-top: 0.8em;padding-top: 8px;padding-bottom: 8px;outline: 0px;color: rgb(53, 53, 53);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, "PingFang SC", Cambria, Cochin, Georgia, Times, "Times New Roman", serif;font-size: 16px;text-align: left;text-indent: 0em;word-spacing: 0.8px;background-color: rgb(255, 255, 255);line-height: 1.75;letter-spacing: 0em;margin-bottom: 24px;"> 通过这些技巧，Spring Boot 应用能够在高并发场景下表现出更强的性能和更低的响应延迟。 </section>

引入Disruptor ，系统性能大幅提升！

作者：微信小助手

<section data-tool="mdnice编辑器" data-website="https://www.mdnice.com" style="margin-bottom: 0px;padding-left: 10px;padding-right: 10px;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;width: auto;font-family: Optima, "Microsoft YaHei", PingFangSC-regular, serif;font-size: 16px;color: rgb(0, 0, 0);line-height: 1.5em;word-spacing: 0em;letter-spacing: 0em;word-break: break-word;text-align: left;"> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">是一个很受欢迎的内存消息队列，它源于 <span style="letter-spacing: normal;">LMAX </span>对并发、性能和非阻塞算法的研究。今天一起来学习一下这个消息队列。</span></p> <h2 data-tool="mdnice编辑器" style="border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(239, 112, 96);margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none none solid;border-width: 1px 1px 2px;border-radius: 0px;box-shadow: none;display: flex;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.1em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="font-size: 22px;color: rgb(255, 255, 255);background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(239, 112, 96);line-height: 1.5em;letter-spacing: 0em;align-items: unset;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 3px 3px 0px 0px;box-shadow: none;display: inline-block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;margin-right: 5px;overflow: unset;padding: 3px 10px 1px;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">简介</span></h2> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">对于主流的分布式消息队列来说，一般会包含<span style="letter-spacing: normal;"> Producer、Broker、Consumer</span>、注册中心等模块。比如 <span style="letter-spacing: normal;">RocketMQ </span>架构如下：</span></p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100006568" data-ratio="0.4856278366111952" src="/upload/a92fda774fcaacaf60cc861cb5f19d27.png" data-type="png" data-w="661" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">并不是分布式消息队列，它是一款内存消息队列，因此架构上跟分布式消息队列有很大差别。下面是一张 <span style="letter-spacing: normal;">LMAX </span>使用 <span style="letter-spacing: normal;">Disruptor </span>的案例图：</span></p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100006569" data-ratio="0.6650124069478908" src="/upload/46f24888fd487963438dfbc5b4e3b4d7.png" data-type="png" data-w="806" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">我们介绍一下 <span style="letter-spacing: normal;">Disruptor </span>架构中的核心概念。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.1 Ring Buffer</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">Ring Buffer 通常被认为是 <span style="letter-spacing: normal;">Disruptor </span>的最主要的设计，但是从 3.0 版本开始，<span style="letter-spacing: normal;">Ring Buffer</span> 只负责存储和更新经过 <span style="letter-spacing: normal;">Disruptor </span>的数据。在一些高级的使用场景，它甚至完全可以被用户替换。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.2 Sequence</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">使用 <span style="letter-spacing: normal;">Sequence </span>来识别特定组件的位置。每个 <span style="letter-spacing: normal;">Consumer</span>（也就是事件处理器）都像 <span style="letter-spacing: normal;">Disruptor </span>一样持有一个 <span style="letter-spacing: normal;">Sequence</span>。并发相关的核心代码依赖 <span style="letter-spacing: normal;">Sequence </span>的自增值，因此 <span style="letter-spacing: normal;">Sequence </span>具有跟 <span style="letter-spacing: normal;">AtomicLong </span>相似的特性，事实上唯一的不同就是不同的 <span style="letter-spacing: normal;">Sequence </span>之间不存在伪共享问题。</span></p> <blockquote data-tool="mdnice编辑器" style="margin-top: 20px;margin-bottom: 20px;padding: 10px 10px 10px 20px;border-top: 3px none rgba(0, 0, 0, 0.4);border-bottom: 3px none rgba(0, 0, 0, 0.4);border-right: 3px none rgba(0, 0, 0, 0.4);border-left-color: rgb(239, 112, 96);border-radius: 0px;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(255, 249, 249);width: auto;height: auto;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;overflow: auto;"> <p style="text-indent: 0em;padding-top: 8px;padding-bottom: 8px;color: rgb(0, 0, 0);line-height: 1.8em;letter-spacing: 0px;"><span style="letter-spacing: 1px;">伪共享：<span style="letter-spacing: normal;">CPU </span>缓存是以缓存行为单位进行加载和存储，<span style="letter-spacing: normal;">CPU </span>每次从主存中拉取数据时，会把相邻的数据也存入同一个缓存行。即使多个线程操作的是同一缓存行中不同的变量，只要有一个线程修改了缓存行中的某一个变量值，该缓存行就会被标记为无效，需要重新从主从中加载。在多线程环境下，频繁地重新加载缓存行，会严重影响了程序执行效率。</span></p> </blockquote> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.3 Sequencer</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Sequencer </span><span style="letter-spacing: 1px;">是 <span style="letter-spacing: normal;">Disrupter </span>的真正核心，有单个生产者和多个生产者两种实现（<span style="letter-spacing: normal;">SingleProducerSequencer </span>和 <span style="letter-spacing: normal;">MultiProducerSequencer</span>）。为了让数据在生产者和消费者之间快速、准确地传输，它们都实现了所有并发算法。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.4 Sequence Barrier</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Sequencer </span><span style="letter-spacing: 1px;">生成一个<span style="letter-spacing: normal;"> Sequence Barrier</span>，它包含由 <span style="letter-spacing: normal;">Sequencer </span>生成的 <span style="letter-spacing: normal;">Sequence </span>和消费者拥有的 <span style="letter-spacing: normal;">Sequence </span>的引用。<span style="letter-spacing: normal;">Sequence Barrier </span>决定是否有事件给消费者处理。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.5 Wait Strategy</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">消费者怎样等待事件的到来。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.6 Event Processor</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">主要负责循环处理来自 <span style="letter-spacing: normal;">Disruptor </span>事件，它拥有消费者 <span style="letter-spacing: normal;">Sequence </span>的所有权。有一个单独的实现类 <span style="letter-spacing: normal;">BatchEventProcessor</span>，这个类拥有高效的事件循环处理能力并且处理完成后可以回调实现 <span style="letter-spacing: normal;">EventHandler </span>接口的用户。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">1.7 Event Handler</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">由用户来实现并且代表 <span style="letter-spacing: normal;">Disruptor </span>消费者的接口。</span></p> <h2 data-tool="mdnice编辑器" style="border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(239, 112, 96);margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none none solid;border-width: 1px 1px 2px;border-radius: 0px;box-shadow: none;display: flex;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.1em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="font-size: 22px;color: rgb(255, 255, 255);background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(239, 112, 96);line-height: 1.5em;letter-spacing: 0em;align-items: unset;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 3px 3px 0px 0px;box-shadow: none;display: inline-block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;margin-right: 5px;overflow: unset;padding: 3px 10px 1px;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">2 Disruptor 特性</span></h2> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2.1 多播事件</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">多播事件是 <span style="letter-spacing: normal;">Disruptor </span>区别于其他队列的最大差异。其他队列都是一个事件消息只能被单个消费者消费，而 <span style="letter-spacing: normal;">Disruptor </span>如果有多个消费者监听，则可以将所有事件消息发送给所有消费者。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">在前面 <span style="letter-spacing: normal;">LMAX </span>使用 <span style="letter-spacing: normal;">Disruptor </span>的案例图中，有 <span style="letter-spacing: normal;">JournalConsumer</span>、<span style="letter-spacing: normal;">ReplicationConsumer </span>和 <span style="letter-spacing: normal;">ApplicationConsumer </span>三个消费者监听了 <span style="letter-spacing: normal;">Disruptor</span>，这三个消费者将收到来了 <span style="letter-spacing: normal;">Disruptor </span>的所有消息。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2.2 消费者依赖关系图</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">为了支持并发处理在实际业务场景中的需要，有时消费者直接需要做协调。再回到前面 <span style="letter-spacing: normal;">LMAX </span>使用 <span style="letter-spacing: normal;">Disruptor </span>的案例，在 <span style="letter-spacing: normal;">journalling </span>和 <span style="letter-spacing: normal;">replication </span>这两个消费者处理完成之前，有必要阻止业务逻辑消费者开始处理。我们称这个特征为“<span style="letter-spacing: normal;">gating</span>”（或者更准确地说，该特征是“gating”的一种形式）。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">首先，确保生产者数量不会超过消费者。这通过调用 <span style="letter-spacing: normal;">RingBuffer.addGatingConsumers（）</span>来将相关消费者添加到 <span style="letter-spacing: normal;">Disruptor</span>。其次，消费者依赖关系的实现是通过构建一个 <span style="letter-spacing: normal;">SequenceBarrier</span>，<span style="letter-spacing: normal;">SequenceBarrier </span>拥有需要在它前面完成处理逻辑的消费者的 <span style="letter-spacing: normal;">Sequence</span>。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">就拿前面 L<span style="letter-spacing: normal;">MAX </span>使用 <span style="letter-spacing: normal;">Disruptor </span>的案例来说，<span style="letter-spacing: normal;">ApplicationConsumer </span>的 <span style="letter-spacing: normal;">SequenceBarrier </span>拥有 <span style="letter-spacing: normal;">JournalConsumer </span>和 <span style="letter-spacing: normal;">ReplicationConsumer </span>这 2 个消费者的 <span style="letter-spacing: normal;">Sequence</span>，所以 <span style="letter-spacing: normal;">ApplicationConsumer </span>对 <span style="letter-spacing: normal;">JournalConsumer </span>和 <span style="letter-spacing: normal;">ReplicationConsumer </span>的依赖关系可以从 <span style="letter-spacing: normal;">SequenceBarrier </span>到 <span style="letter-spacing: normal;">Sequence </span>的连接中看到。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Sequencer </span><span style="letter-spacing: 1px;">和下游消费者的关系也需要注意。<span style="letter-spacing: normal;">Sequencer </span>的一个角色就是发布的事件消息不能超出<span style="letter-spacing: normal;"> Ring Buffer</span>。这就要求下游消费者的 <span style="letter-spacing: normal;">Sequence </span>不能小于<span style="letter-spacing: normal;"> Ring Buffer </span>的<span style="letter-spacing: normal;"> Sequence</span>，也不能小于<span style="letter-spacing: normal;"> Ring Buffer </span>的大小。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">上面图中，因为<span style="letter-spacing: normal;"> ApplicationConsumer </span>的<span style="letter-spacing: normal;"> Sequence </span>必须要保证小于等于 <span style="letter-spacing: normal;">JournalConsumer </span>和<span style="letter-spacing: normal;"> ReplicationConsumer </span>的<span style="letter-spacing: normal;"> Sequence</span>，因此 <span style="letter-spacing: normal;">Sequencer </span>只需要关心<span style="letter-spacing: normal;"> ApplicationConsumer </span>的<span style="letter-spacing: normal;"> Sequence</span>。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2.3 内存预分配</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">的目标是低延迟，因此减少或者去除内存分配是必要的。在基于 <span style="letter-spacing: normal;">Java </span>的系统中，目标是减少 <span style="letter-spacing: normal;">STW </span>次数。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">为了支持这一点，用户可以在 <span style="letter-spacing: normal;">Disruptor </span>中预分配事件所需的内存。在预分配内存时，用户提供的<span style="letter-spacing: normal;"> EventFactory </span>将对<span style="letter-spacing: normal;"> Ring Buffer </span>的所有元素进行调用。当生产者向<span style="letter-spacing: normal;"> Disruptor </span>发送新的事件消息时，<span style="letter-spacing: normal;">Disruptor </span>的<span style="letter-spacing: normal;"> API </span>允许用户使用构造好的对象，他们可以调用对象的方法或者更新对象的字段。<span style="letter-spacing: normal;">Disruptor </span>需要确保并发安全。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">2.4 无锁并发</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">Disruptor </span><span style="letter-spacing: 1px;">实现低延迟的另一个关键方法时使用无锁算法，通过使用内存屏障和<span style="letter-spacing: normal;"> CAS </span>来实现内存可见性和正确性。<span style="letter-spacing: normal;">Disruptor </span>唯一使用锁的地方就是在<span style="letter-spacing: normal;"> BlockingWaitStrategy</span>。</span></p> <h2 data-tool="mdnice编辑器" style="border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(239, 112, 96);margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none none solid;border-width: 1px 1px 2px;border-radius: 0px;box-shadow: none;display: flex;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.1em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="font-size: 22px;color: rgb(255, 255, 255);background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(239, 112, 96);line-height: 1.5em;letter-spacing: 0em;align-items: unset;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 3px 3px 0px 0px;box-shadow: none;display: inline-block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;margin-right: 5px;overflow: unset;padding: 3px 10px 1px;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">3 调优选项</span></h2> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">虽然大多数场景下<span style="letter-spacing: normal;"> Disruptor </span>可以表现出优秀的性能，但是仍然有一些调优参数可以改进<span style="letter-spacing: normal;"> Disruptor </span>的性能。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">3.1 单个/多个生产者</span></h3> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;">Disruptor<LongEvent> disruptor = <span style="color: #c678dd;line-height: 26px;">new</span> Disruptor(<br> factory,<br> bufferSize,<br> DaemonThreadFactory.INSTANCE,<br> ProducerType.SINGLE, <br> <span style="color: #c678dd;line-height: 26px;">new</span> BlockingWaitStrategy() <br>);<br></code></pre> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">上面是<span style="letter-spacing: normal;"> disruptor </span>的构造函数，<span style="letter-spacing: normal;">ProducerType.SINGLE </span>表示创建单生产者的 <span style="letter-spacing: normal;">Sequencer</span>，<span style="letter-spacing: normal;">ProducerType.MULTI  </span>表示创建多生产者的<span style="letter-spacing: normal;"> Sequencer</span>。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">在并发系统中提高系统性能的最好方式是遵循单写原则。下面是官方的一个 <span style="letter-spacing: normal;">disruptor </span>吞吐量测试结果，测试环境是<span style="letter-spacing: normal;"> i7 Sandy Bridge MacBook Air。</span></span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">单生产者：</span></p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100006570" data-ratio="1.0259067357512954" src="/upload/ec8a0f8df318ff27fa65974153968463.jpg" data-type="jpeg" data-w="386" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">多生产者：</span></p> <figure data-tool="mdnice编辑器" style="margin-top: 10px;margin-bottom: 10px;display: flex;flex-direction: column;justify-content: center;align-items: center;"> <img class="rich_pages wxw-img" data-imgfileid="100006571" data-ratio="1.0335917312661498" src="/upload/2116b2504923610e283a8576edd9b022.jpg" data-type="jpeg" data-w="387" style="display: block;margin-right: auto;margin-left: auto;border-style: none;border-width: 3px;border-color: rgba(0, 0, 0, 0.4);border-radius: 0px;object-fit: fill;box-shadow: rgba(0, 0, 0, 0) 0px 0px 0px 0px;"> </figure> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">3.2 等待策略</span></h3> <ol data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: normal;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: normal;">BlockingWaitStrategy</span> <span style="letter-spacing: 1px;"></span> </section></li> </ol> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">disruptor </span><span style="letter-spacing: 1px;">的默认等待策略是 <span style="letter-spacing: normal;">BlockingWaitStrategy</span>，这种策略使用锁和唤醒锁的 <span style="letter-spacing: normal;">Condition </span>变量。</span></p> <ol start="2" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: normal;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: normal;">SleepingWaitStrategy</span> <span style="letter-spacing: 1px;"></span> </section></li> </ol> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">跟 <span style="letter-spacing: normal;">BlockingWaitStrategy </span>策略类似，他是通过 <span style="letter-spacing: normal;">LockSupport.parkNanos(1) </span>方法来实现等待，不需要给 <span style="letter-spacing: normal;">Condition </span>变量发送信号来唤醒等待。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">主要适用于对延时要求不高的场景，比如异步打印日志。</span></p> <ol start="3" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: normal;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: normal;">YieldingWaitStrategy</span> <span style="letter-spacing: 1px;"></span> </section></li> </ol> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">‌<span style="letter-spacing: normal;">YieldingWaitStrategy </span>策略使用<span style="letter-spacing: normal;"> Busy spin‌</span>（不释放 <span style="letter-spacing: normal;">CPU </span>资源，通过循环检查条件直到条件满足为止）技术来等待 <span style="letter-spacing: normal;">sequence </span>增长到一个合适的值。在循环内部会调用 Thread#yield() 方法允许其他排队线程去执行。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">这种策略主要用于通过消耗 <span style="letter-spacing: normal;">CPU </span>来实现低延迟的场景。当 <span style="letter-spacing: normal;">EventHandler </span>数量消息逻辑 <span style="letter-spacing: normal;">CPU </span>核数并且对延迟要求较高时，可以考虑这种等待策略。</span></p> <ol start="4" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: normal;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: normal;">BusySpinWaitStrategy</span> <span style="letter-spacing: 1px;"></span> </section></li> </ol> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: normal;">BusySpinWaitStrategy </span><span style="letter-spacing: 1px;">是性能最高的等待策略，它适用于低延迟系统，但是对部署环境要求很高。</span></p> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">这种等待策略的唯一适用场景是当 <span style="letter-spacing: normal;">EventHandler </span>数量消息逻辑 <span style="letter-spacing: normal;">CPU </span>核数并且超线程被禁用。</span></p> <h3 data-tool="mdnice编辑器" style="margin-top: 30px;margin-bottom: 15px;"><span style="font-size: 20px;line-height: 1.5em;letter-spacing: 0em;font-weight: bold;display: block;">4 官方示例</span></h3> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">下面是一个官方示例。这个例子比较简单，就是生产者向消费者发送一个 <span style="letter-spacing: normal;">long </span>类型的值。</span></p> <ol data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: 1px;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: 1px;">首先定义一个 <span style="letter-spacing: normal;">Event</span>。</span> </section></li> </ol> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">public</span> <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span> <span style="color: #e6c07b;line-height: 26px;">LongEvent</span><br></span>{<br>    <span style="color: #c678dd;line-height: 26px;">private</span> <span style="color: #c678dd;line-height: 26px;">long</span> value;<br><br>    <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span> <span style="color: #c678dd;line-height: 26px;">void</span> <span style="color: #61aeee;line-height: 26px;">set</span><span style="line-height: 26px;">(<span style="color: #c678dd;line-height: 26px;">long</span> value)</span><br>    </span>{<br>        <span style="color: #c678dd;line-height: 26px;">this</span>.value = value;<br>    }<br><br>    <span style="color: #61aeee;line-height: 26px;">@Override</span><br>    <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span> String <span style="color: #61aeee;line-height: 26px;">toString</span><span style="line-height: 26px;">()</span><br>    </span>{<br>        <span style="color: #c678dd;line-height: 26px;">return</span> <span style="color: #98c379;line-height: 26px;">"LongEvent{"</span> + <span style="color: #98c379;line-height: 26px;">"value="</span> + value + <span style="color: #98c379;line-height: 26px;">'}'</span>;<br>    }<br>}<br></code></pre> <ol start="2" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: 1px;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: 1px;">为了能让 <span style="letter-spacing: normal;">Disruptor </span>预分配内存，这里定义一个 <span style="letter-spacing: normal;">LongEventFactory</span>。</span> </section></li> </ol> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">public</span> <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span> <span style="color: #e6c07b;line-height: 26px;">LongEventFactory</span> <span style="color: #c678dd;line-height: 26px;">implements</span> <span style="color: #e6c07b;line-height: 26px;">EventFactory</span><<span style="color: #e6c07b;line-height: 26px;">LongEvent</span>><br></span>{<br>    <span style="color: #61aeee;line-height: 26px;">@Override</span><br>    <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span> LongEvent <span style="color: #61aeee;line-height: 26px;">newInstance</span><span style="line-height: 26px;">()</span><br>    </span>{<br>        <span style="color: #c678dd;line-height: 26px;">return</span> <span style="color: #c678dd;line-height: 26px;">new</span> LongEvent();<br>    }<br>}<br></code></pre> <ol start="3" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: 1px;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: 1px;">创建一个消费者来处理事件</span> </section></li> </ol> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">public</span> <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span> <span style="color: #e6c07b;line-height: 26px;">LongEventHandler</span> <span style="color: #c678dd;line-height: 26px;">implements</span> <span style="color: #e6c07b;line-height: 26px;">EventHandler</span><<span style="color: #e6c07b;line-height: 26px;">LongEvent</span>><br></span>{<br>    <span style="color: #61aeee;line-height: 26px;">@Override</span><br>    <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span> <span style="color: #c678dd;line-height: 26px;">void</span> <span style="color: #61aeee;line-height: 26px;">onEvent</span><span style="line-height: 26px;">(LongEvent event, <span style="color: #c678dd;line-height: 26px;">long</span> sequence, <span style="color: #c678dd;line-height: 26px;">boolean</span> endOfBatch)</span><br>    </span>{<br>        System.out.println(<span style="color: #98c379;line-height: 26px;">"Event: "</span> + event);<br>    }<br>}<br></code></pre> <ol start="4" data-tool="mdnice编辑器" style="margin-top: 8px;margin-bottom: 8px;padding-left: 25px;" class="list-paddingleft-1"> <li style="letter-spacing: 1px;"> <section style="margin-top: 5px;margin-bottom: 5px;color: rgb(1, 1, 1);line-height: 1.8em;letter-spacing: 0em;"> <span style="letter-spacing: 1px;">编写发送事件消息的逻辑</span> </section></li> </ol> <pre data-tool="mdnice编辑器" style="border-radius: 5px;box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;margin-top: 10px;margin-bottom: 10px;"><code style="overflow-x: auto;padding: 16px;color: #abb2bf;padding-top: 15px;background: #282c34;border-radius: 5px;display: -webkit-box;font-family: Consolas, Monaco, Menlo, monospace;font-size: 12px;"><span style="color: #c678dd;line-height: 26px;">import</span> com.lmax.disruptor.dsl.Disruptor;<br><span style="color: #c678dd;line-height: 26px;">import</span> com.lmax.disruptor.RingBuffer;<br><span style="color: #c678dd;line-height: 26px;">import</span> com.lmax.disruptor.examples.longevent.LongEvent;<br><span style="color: #c678dd;line-height: 26px;">import</span> com.lmax.disruptor.util.DaemonThreadFactory;<br><span style="color: #c678dd;line-height: 26px;">import</span> java.nio.ByteBuffer;<br><br><span style="color: #c678dd;line-height: 26px;">public</span> <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">class</span> <span style="color: #e6c07b;line-height: 26px;">LongEventMain</span><br></span>{<br>    <span style="line-height: 26px;"><span style="color: #c678dd;line-height: 26px;">public</span> <span style="color: #c678dd;line-height: 26px;">static</span> <span style="color: #c678dd;line-height: 26px;">void</span> <span style="color: #61aeee;line-height: 26px;">main</span><span style="line-height: 26px;">(String[] args)</span> <span style="color: #c678dd;line-height: 26px;">throws</span> Exception<br>    </span>{<br>        <span style="color: #c678dd;line-height: 26px;">int</span> bufferSize = <span style="color: #d19a66;line-height: 26px;">1024</span>; <br><br>        Disruptor<LongEvent> disruptor = <br>                <span style="color: #c678dd;line-height: 26px;">new</span> Disruptor<>(LongEvent::<span style="color: #c678dd;line-height: 26px;">new</span>, bufferSize, DaemonThreadFactory.INSTANCE);<br><br>        disruptor.handleEventsWith((event, sequence, endOfBatch) -><br>                System.out.println(<span style="color: #98c379;line-height: 26px;">"Event: "</span> + event)); <br>        disruptor.start(); <br><br><br>        RingBuffer<LongEvent> ringBuffer = disruptor.getRingBuffer(); <br>        ByteBuffer bb = ByteBuffer.allocate(<span style="color: #d19a66;line-height: 26px;">8</span>);<br>        <span style="color: #c678dd;line-height: 26px;">for</span> (<span style="color: #c678dd;line-height: 26px;">long</span> l = <span style="color: #d19a66;line-height: 26px;">0</span>; <span style="color: #c678dd;line-height: 26px;">true</span>; l++)<br>        {<br>            bb.putLong(<span style="color: #d19a66;line-height: 26px;">0</span>, l);<br>            ringBuffer.publishEvent((event, sequence, buffer) -> event.set(buffer.getLong(<span style="color: #d19a66;line-height: 26px;">0</span>)), bb);<br>            Thread.sleep(<span style="color: #d19a66;line-height: 26px;">1000</span>);<br>        }<br>    }<br>}<br></code></pre> <h2 data-tool="mdnice编辑器" style="border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(239, 112, 96);margin-top: 30px;margin-bottom: 15px;align-items: unset;background-attachment: scroll;background-clip: border-box;background-image: none;background-origin: padding-box;background-position: 0% 0%;background-repeat: no-repeat;background-size: auto;border-style: none none solid;border-width: 1px 1px 2px;border-radius: 0px;box-shadow: none;display: flex;flex-direction: unset;float: unset;height: auto;justify-content: unset;line-height: 1.1em;overflow: unset;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;"><span style="font-size: 22px;color: rgb(255, 255, 255);background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(239, 112, 96);line-height: 1.5em;letter-spacing: 0em;align-items: unset;border-style: none;border-width: 1px;border-color: rgb(0, 0, 0);border-radius: 3px 3px 0px 0px;box-shadow: none;display: inline-block;font-weight: bold;flex-direction: unset;float: unset;height: auto;justify-content: unset;margin-right: 5px;overflow: unset;padding: 3px 10px 1px;text-indent: 0em;text-shadow: none;transform: none;width: auto;-webkit-box-reflect: unset;">5 总结</span></h2> <p data-tool="mdnice编辑器" style="line-height: 1.8em;letter-spacing: 0em;text-indent: 0em;padding-top: 8px;padding-bottom: 8px;"><span style="letter-spacing: 1px;">作为一款高性能的内存队列，有不少优秀的设计思想值得我们学习，比如内存预分配、无锁并发。同时它的使用非常简单，推荐大家使用。</span></p> </section>

文章列表