Repository navigation
Conversation
Member
|
"Compile unit" is a great idea. We need to modularise the compiling process rather than hard code everything in the main deploy function. Ideally, each compiling module declares its dependencies, and the main deploy function only executes the constructed graph of compile units. Wait, isn't that similar to what I want to achieve in |
Contributor
Author
|
好像还有点东西要改我还没推，资源预估上的 |
WorkspaceUpdate now collects schema compile units serially (phase 1), then compiles independent dictionaries concurrently (phase 2). Units whose output files overlap (shared dictionaries, packs, prisms) fall back to serial compilation. Batching is constrained by a memory budget (1/4 of physical memory, estimated per dictionary from source size) and a concurrency cap (min(hardware_concurrency, 8)); oversized dictionaries are compiled alone. Compilation exceptions are caught and counted as failures. ENABLE_THREADING=OFF keeps the original serial path via RIME_NO_THREADING.
Move schema/dependencies processing out of process_schema and into the top-level loop, so a schema's own dictionary is compiled before its direct dependencies, matching upstream serial build order. Only one level of dependencies is processed; nested dependencies are not compiled.
…error handling - Reduce kMaxConcurrency from 8 to 4 to limit memory usage on mobile devices - Lower memory budget from physical_memory/4 to physical_memory/8 - Reduce fallback budget from 2GB to 512MB for constrained environments - Increase kEstimatedBytesPerSourceByte from 16 to 32 for more accurate estimation - Add std::bad_alloc catch in compile_unit for graceful OOM handling - Wrap future.get() in try-catch to prevent std::terminate from escaped exceptions
Extract CompileUnit, CompileGraph and CompileScheduler from WorkspaceUpdate into rime/lever/compile_unit. Each compiling module declares its dependencies, output files and memory estimate; the deploy function only constructs the graph and executes it. Units sharing an output file are chained in declaration order instead of falling back to a fully serial path, so independent units stay parallel.
SchemaUpdate now goes through BuildDictionaryCompileUnit and the CompileScheduler like WorkspaceUpdate, dropping the duplicated dictionary creation and compile invocation.
ksqsf
reviewed
Oct 31, 2026
ksqsf
left a comment
Member
There was a problem hiding this comment.
Overall LGTM. Some minor comments.
ksqsf
reviewed
Oct 31, 2026
| * the in-flight set may not exceed 1/8 of detected memory (512 MB fallback if | ||
| the limit cannot be detected); | ||
| * detected memory is the smaller of physical memory and the Linux cgroup limit; | ||
| * concurrency is ≤ min(hardware cores, 4); |
Member
|
很好很強大。強烈支持。 |
- Rename BuildDictionaryCompileUnit to CreateDictionaryCompileUnit - Rename CompileUnitResult enum values for clarity: - kBuilt -> kCreated - kNoDictionary -> kTrivial - Simplify MaybeCreateDirectory by removing redundant fs::exists check - Move MaybeCreateDirectory closer to its original position to reduce diff - Change CompileScheduler::Run to return std::optional<Result> - Update all call sites and tests Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
|
改好了，如合并记得去掉那个跑分提交 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request
Issue tracker
Fixes will automatically close the related issue
Fixes #
Feature
Describe feature of pull request
Unit test
Manual test
Code Review
Additional Info
实现方式
1. 编译单元与编译图
新增
src/rime/lever/compile_unit.{h,cc}：CompileUnit描述一个编译任务：id：通常为schema_id；prerequisites：必须先完成的单元；outputs：该单元可能写入的文件（不含扩展名）；estimated_memory：用于调度估算的峰值内存；run：实际编译动作。CompileGraph负责收集单元并维护顺序：AddDependency(prerequisite, dependent)表达；outputs的单元按声明顺序串行化，避免并发写同一个 table/prism/reverse 文件；AddUnit/AddDependency的失败不再静默，调用方会记录日志并计入失败。CompileScheduler负责执行：ENABLE_THREADING=OFF（RIME_NO_THREADING）时顺序执行，保持可回退路径；2. 两阶段部署
WorkspaceUpdate仍然保持“先收集、后执行”的两阶段结构：Dictionary对象、访问组件缓存，并构造CompileUnit。SchemaUpdate与WorkspaceUpdate复用BuildDictionaryCompileUnit，避免两套词典编译逻辑。3. 资源限制
并行度同时受 CPU 和内存约束：
min(hardware_concurrency, 4)；std::bad_alloc和普通异常，单个单元失败不会导致进程崩溃。4. 生命周期与所有权
BuildDictionaryCompileUnit改为接收the<Config>，调用方不再使用裸指针release()；CompileUnit的依赖和输出只能通过受控接口修改，减少绕过图约束的可能；AddUnit/AddDependency的返回值在调用点被检查，避免单元或顺序约束被静默丢弃。风险与处理
CompileUnit声明outputs；写同一输出的单元按声明顺序串行化；benchmark 校验冷启动产物逐字节一致Dictionary构造；Phase 2 只执行已构造单元bad_alloc兜底RIME_NO_THREADING现在也走统一调度器，而不是原来的WorkspaceUpdate串行循环ENABLE_THREADING=OFF作为回退开关WorkspaceUpdate最终返回failure == 0；独立单元继续执行，避免一个坏方案阻塞全部部署；单元失败会记录具体 idAddDependency在冲突时保留声明顺序，依赖请求会被忽略join退出；单元测试覆盖依赖顺序、输出串行化和失败隔离CompileUnit单元测试；Linux benchmark CI 将 6 个生成方案与模拟或真实kongmingma方案混合部署，并检查产物一致性验证
单元测试
结果：全部通过。新增
RimeCompileUnitTest共 8 个用例，覆盖：部署 benchmark
环境：8 核 / 7.7 GB，合成 6 个方案、每个 15 万词条。
多次运行的 cold 加速比约在 2.0–2.3×。冷启动产物在 serial 与 parallel 之间逐字节一致。
“单本词典改动”场景基本持平，符合预期：只有一个词典需要重编译，并行度无法发挥作用，该场景主要用于防止回归。
引入主线程回退后复测：cold
15624 ms → 7230 ms（2.16×），warm 2.54×，单本改动 1.04×，产物仍逐字节一致。真实大词库 benchmark
环境：8 核 / 7.7 GB。数据：
kongmingma（11 MB，77 万行）+kongmingmabig（78 MB，422 万行），按此顺序部署。默认
kMemoryBudgetDivisor=8时，kongmingmabig的估算内存（约 2.5 GB）超过预算（约 1 GB），调度器只允许它单独执行。因此并行构建下两个单元实际仍按小、大顺序串行：峰值 RSS：old 约 2.44 GB，new 约 2.85 GB。冷启动产物逐字节一致。本机 old 峰值未复现 >3.2 GB，可能与插件或词典结构有关。
调参实验：临时把预算改为
memory_limit / 2（本机约 4 GB）时，两个单元可以同时启动；相邻测量 cold84.0 s（/8）→73.7 s（/2），峰值2.86 GB→2.79 GB。大词典自身因资源争用从约 69 s 放慢到约 74 s，净收益约 10%。4 GB 设备上该策略的预算为 2 GB，超大词典仍会独占，但会提高中等词典并发的 OOM 风险，暂未采用；当前保持/8。加入主线程回退后，/8 路径重新退化为与 old 相同的执行线程：峰值降至
2.44 GB，cold 用时80.7 s。混合方案 benchmark 与 Linux CI
benchmark-only 提交跟踪了真实空明码
kongmingma/kongmingmabig方案和词典（约 89 MB），并添加 Linux CI。该提交标记为[don't merge this]，仅用于跑分验证，不进入上游。CI 使用同一数据集混合 6 个生成测试方案（每个 15 万条）和两本空明码方案，分别测试：CI 对比串行与并行冷启动产物，发现差异即失败，并上传跑分报告。Linux 本机混合测试结果如下；warm / modified 详细数据见
benchmarks/deploy/report-kongmingma.md：两种负载的串行与并行冷启动产物均逐字节一致。真实词典场景的加速有限，主要受大词典单本编译时间和
/8内存预算约束。回退方式
-DENABLE_THREADING=OFF即回到顺序执行路径；后续可做
/8对超大词典会安全退化为串行；