In this post, I would like to cover my experiments with Apple Intelligence – the on-device, built-in language model – specifically focusing on extracting data (in this case, dates) from user voice input: Voice → Speech-to-Text → LM → App API → App GUI.
Before I start, I would like to make my position on LLMs clear. I do not see much sense or value in generative AI overall, except when it comes to working with language (f.e. proofreading) and generating code snippets. However, I do see a future for LLMs as local models used as a communication interface/level between a user and an application API. Take everything after this with a grain of salt – I’m a skeptic
Let’s start with the problems I encountered:
– The built-in model (Apple Intelligence) is available in limited regions.
– It supports a very limited number of languages.
– The model is tiny and highly limited in the tasks it can perform.
– The context window is tiny – 4,096 tokens, which greatly limits instructions, tool count, and conversation length.
– Performance: For my usage, I need to get a full LM response with a generated data structure. Even with a tiny model, a narrow context window, simple instructions, and a minimal number of tools, the response can take 1 to 3 seconds. This is hard to hide from the user and feels somewhat sluggish.
– Unpredictable, ever-changing results for some requests due to the random seed nature of LLM sessions.
Is the “Speech-to-Text to LLM to Data” approach a good one? Not really, at least not yet (and I doubt it will change much due to the inherent nature of LLMs and the resource limitations of mobile devices). It’s way too unpredictable, unstable, and sluggish.
As I see it, the only practical way to use this approach is as a fallback for “scripted” options:
User Voice Input → Speech Recognition → Traditional Pattern Matching & Parsing → LM (if the previous step fails) → App API → UI
This way, we can handle all common patterns a user might input for entering a date – with the precision and repeatability of traditional software, and If a user enters something unusual or uncommon, we can give the LLM a chance to work its magic. In those cases, a 70–80% success rate is a reasonable result.
If you would like to play around with this approach yourself, you can find a minimal playground here: https://github.com/AndreiMaksimovich/SwiftDeveloperBlog/tree/main/07%20-%20Foundation%20Model/FoundationModel
