Happy to share some of our experiences at TovTech working with language models over the past year, where the main uses have been data analysis and developing solutions in Hebrew. There’s a lot of documentation on the internet about using language models for data analysis, but almost none about working in Hebrew, so I’ll try to share our experience in this area.
Let’s start from the starting point of November 2022 and the launch of ChatGPT. Its abilities in Hebrew were about the same as Google Translate, and there wasn’t really much to work with.
By March 2023, the release of Claude and GPT-4 already showed that language models in Hebrew are much more than Google Translate. The phrasing was already at a high level, and there were much fewer syntax errors.
May 2023 saw the release of Google’s Palm 2, and not everyone may agree with me, but in my opinion, it’s the best model for Hebrew that has been released. In several comparisons we did, it achieved the best results (conversation management, creation, extracting data from text).
December 2023 saw the release of Gemini, which improves Palm 2’s capabilities in Hebrew. Despite the improvements, we see that Gemini is still not stable enough and can sometimes write texts in Chinese, for example. In addition, there’s still no option for context, another factor leading to instability.
The additional reason we decided to use Google’s models is the pricing:
Claude 2: Price for 1M tokens: Input: $8 Output: $24
GPT-4–1106-preview: Price for 1M tokens: Input: $10 Output: $30
Gemini Pro: Price for 1M characters: Input: $0.25 Output: $0.5
The different pricing of Google makes the comparison a bit difficult, but assuming that every 4 chars are a token, Google’s pricing is 80%-90% cheaper. Additionally, Google allows the use of GCP credits for its language models, unlike Amazon, which does not allow using its credits for external services like Claude.
I’d love to hear more insights or questions you have about using language models in Hebrew.
If you want to see more posts about our work at Tov-Tech, feel free to follow us on LinkedIn: https://www.linkedin.com/in/raz-hadas/
