Repository navigation
Enhance voice transcription, image generation, and attachment handling - #2
Merged
Merged
Conversation
…ge description via SmolVLM with support for transient engine management.
…ts and token limits, and replace confirmationDialog with sheet for photo attachments
…writer rendering during generation
…nd StreamChunkView for consistent block processing.
…ith RAM-efficient model swapping
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughPhase 3 adds Whisper (speech-to-text), SmolVLM (vision description), and Stable Diffusion (image generation) as on-demand specialist sidecars. New C++ bridges and xcframework build scripts expose these libraries to Swift through updated headers and an Xcode project. A specialist catalog, download manager, and Swift wrappers integrate the sidecars into ChangesPhase 3 specialist sidecar integration
Sequence Diagram(s)sequenceDiagram
actor User
participant ChatView
participant ChatSession
participant resolveMedia as ChatSession.resolveMedia
participant WhisperTranscriber
participant VisionDescriber
participant ImageGenerator
participant InferenceEngine
User->>ChatView: tap send with audio/image attachment
ChatView->>ChatSession: generate()
ChatSession->>resolveMedia: resolveMedia(attachments, userText)
alt audio attachment, no native support
resolveMedia->>WhisperTranscriber: transcribe(fileURL:languageCode:)
WhisperTranscriber-->>resolveMedia: transcript String
else image attachment, no native support
resolveMedia->>VisionDescriber: describe(imageData:userQuestion:)
VisionDescriber->>InferenceEngine: load SmolVLM, stream, unload
InferenceEngine-->>VisionDescriber: description text
VisionDescriber-->>resolveMedia: description String
end
resolveMedia-->>ChatSession: ResolvedMedia(directMedia, promptText)
ChatSession->>InferenceEngine: generate with resolved media/prompt
User->>ChatView: tap "Create with AI"
ChatView->>ChatSession: createImage(prompt:)
ChatSession->>InferenceEngine: unload main model
ChatSession->>ImageGenerator: generate(prompt:negativePrompt:)
ImageGenerator-->>ChatSession: JPEG Data
ChatSession->>InferenceEngine: reload main model
ChatSession-->>ChatView: new assistant message with image attachment
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces support for two new native on-device specialists: Whisper (speech-to-text) and Stable Diffusion (image generation), alongside improvements to the core engine's resource management and project structure. The changes add new C/C++ bridges for these specialists, integrate their frameworks into the Xcode project, and update documentation and Swift interfaces to expose their capabilities.
Specialist Integration and Bridging:
synap_whisper.h/cpp) and Stable Diffusion (synap_sd.h/cpp), enabling on-device speech-to-text and image generation via pure-C APIs for Swift interoperability. [1] [2] [3] [4]SynapLink-Bridging-Header.h) to import the new specialist headers.Project and Build System Updates:
whisper.cppandstable-diffusion.cppas git submodules, and integrated their frameworks (whisper.xcframework,sd.xcframework) into the Xcode project file, including build and embed steps. [1] [2] [3] [4] [5] [6]README.mdto reflect the new frameworks, build instructions, and test scripts for all three native engines.Core Engine Improvements:
Swift Interface Enhancements:
ChatSessionclass to expose effective input capabilities (image, audio, media attachment) based on the availability of either the main model or installed specialists.Summary by CodeRabbit
New Features
Documentation