Thoughts on future version 2.0

[ZhymabekRoman]

First of all thanks for the library, I now want to rewrite my Discord bot translator using this library.

After analyzing the entire library code, I thought - why don't we completely refactor the code and make a new version. After a few minutes of thinking, I decided to look at all the fork repositories, maybe there is something not ordinary, and found a commit of one person - diogosilva30@a104f6a . The person completely redone the code. The only one there is no tests and the old implementation of the Yandex translator.

It will be interesting to listen your thoughts on this matter

[Animenosekai]

Hey @ZhymabekRoman I obviously looked at the code of diogosilva30/translate@a104f6a but I decided not to do anything for now with it because there is simply nothing left on there.

The only thing that is left are the translation methods, which is far less than the current features: obviously translation but also transliteration, language detection, text to speech, etc.

Also, he removed the whole Language class part, which can't be implemented in translatepy as it would break too much things.

But what I think we could add from his fork is:

• A language check to see if the service supports the given source or destination language.
• A for loop to check the different services (already implemented on translatepy tho)
• Exception Raising
• A better class management, with base classes (and maybe a class per Translator function?)
• A "cleaner" code?

[ZhymabekRoman]

The only thing that is left are the translation methods, which is far less than the current features: obviously translation but also transliteration, language detection, text to speech, etc.

Oh, yeah, I didn't notice that.

[Animenosekai]

Also, I plan on adding (not on the fork):

• Proxy support
• The MyMemory translation API (+1 service on the services list)
• Your YandexTranslate implementation (or maybe in v1.7? Let me know)

[ZhymabekRoman]

• Your YandexTranslate implementation (or maybe in v1.7? Let me know)

Yes, I think it can already be added in v1.7

[ZhymabekRoman]

I have some plans, too:

• Reimplement (or just fix) Reverso Translate, at the moment it does not work for me at all

Screenshot

• The PROMT translation API (yet +1 service on the services list)
• Write tests that will check the operability of each Translation service, I think it can be implemented now

[Animenosekai]

I think this is because Reverso uses 3 letters ISO language code

[ZhymabekRoman]

I understood why Reverso did not want to work - I set the language of translation ru, turns out need to specify rus. Need to fix this in v2

[Animenosekai]

It works well with the Translator() class in the current version

[ZhymabekRoman]

In general, I will try to implement v2 from scratch myself, I will periodically inform you about the progress of work. I got a picture of a new code in my head

[ZhymabekRoman]

So, I started working on a new version of the library, I thought that I would not really change anything radically, but it turned out the opposite. I must say right away, perhaps the old code somewhere surpasses my own, but I personally did not set out to rewrite the entire code, I just wanted to do refactoring so that in the future it would be easier to add new translation services. At the moment, the code does not work correctly, the only thing that works in it is the Language class and partially the implementation of the Yandex translator (Upd: I accidentally broke it with the last commit). Since I have a little time problems at the moment, I want to leave some notes about the upcoming changes in version v2, within a month I promise to bring the code to perfection.

1. Completely changed the mechanism for storing language data in the iso639 standard, if previously all language data was stored in separate json files, now all data is stored in a single script body as a named typle. It seems to me that this approach allows you not to duplicate data in different json files and in theory should speed up the work. All the data about the language in the iso639 standard was generated from json files that were in the 'data' folder.

2. Due to the changes in the method of storing data about the language in the iso639 standard, the Language class itself has also changed: the multi-level if-else blocks have been removed, and the fuzuwuzyy library is used in an inaccurate comparison by the name of the language, and much more. And by the way, the Language class is backwards compatible with v1.6.

3. Abandoning global variables for caching, in the future - switching to a decorator that will be responsible for caching. At the moment, I am writing a special decorator that will cache the values in the LRU algorithm. I tried the lru_cache decorator from functiontools - it doesn't fulfill the main criteria at all - caching class methods between multiple instances of the class.

.... and also a lot of things. Here is the experimental code itself: main...ZhymabekRoman:main

[Animenosekai]

Hey and thank you for this big commit!

I looked at all of the code and found a lot of interesting things that should definitely be in the v2!!!

Including:

• I knew we eventually need to start raising exceptions when something goes wrong instead of just returning None as if nothing happened lol
• The Language class looks nice and there is indeed no need for a big code block with all of the languages attributes
• Having a separate class for each function returning value is nice!
• Abandoning global caches
• The BaseTranslator class is nice to add new services
• A common request class is very nice to easily implement proxies (and also to verify stuff because requests have had memory leaks issues, don't remember if they fixed it or not but I had a memory problem once with it on a server)!

Also, I have a few questions:

• Why using two classes for the Language class ?
• Do you think if it could be useful to cache the fact that it did not succeeded into getting the language? (Like maybe None for _language and the similarity)
• Maybe we should just return None if there is no such attribute on the Language class
• Should we add a way of getting the best match even with the exception?
• Should we use fuzzywuzzy or the library fuzzywuzzy uses to get the Levinstein Distance?
• Any reason why you are JSON encoding things in iso639.py?
• Do you need any help with the code?

Also there is some minor things to change sometimes but that's a very great job!

Sorry I was quite busy the past days

[ZhymabekRoman]

• Why using two classes for the Language class ?

At the very beginning, I thought to inherit the BaseTranslator class from the BaseLanguage class, but I realized that this is a bad idea, can lead to Spaghetti code, so now we can safely combine the BaseLanguage and Language classes.

• Do you think if it could be useful to cache the fact that it did not succeeded into getting the language? (Like maybe None for _language and the similarity)

I think yes this is a great idea, it should speed up the library, already made changes to the code: ZhymabekRoman@56805ab

• Maybe we should just return None if there is no such attribute on the Language class

It seems to me there is no point in doing this, if a person (like me in the screenshot) enters the wrong language name or language code, then let an exception be raised, the person will know for sure that he or she did not enter the name or code of the language correctly, or such a class attribute does not really exist

• Should we add a way of getting the best match even with the exception?

I unfortunately have no thoughts on this, so I can't answer it. Do you mean that instead of showing one of the most similar languages to show a list of similar languages?

• Should we use fuzzywuzzy or the library fuzzywuzzy uses to get the Levinstein Distance?

It is not necessary to use fuzzywuzzy, we can also use the previous implementation that was used in v1. I just have some experience with this library and I decided to use it, especially since it works from python 2.7+

• Any reason why you are JSON encoding things in iso639.py?

When I generated the file iso639.py from the JSON files in the folder 'data', I faced a problems that caused the interpreter to refuse to run the file. I had to encode it in JSON. We can try again, here is the script itself that generated the iso639 file:

import data
import json

# from icecream import ic  # for debug only


def in_other_languages(alpha2: str):
    for _alpha2, _list in data.LANGUAGES_TO_INTERNATIONAL.items():
        if _alpha2 == alpha2:
            # ic(_list)
            return json.dumps(_list)
    else:
        return None


def alpha3_by_alpha2(alpha2: str):
    for _alpha2, _alpha3 in data.ALPHA2_TO_ALPHA3.items():
        if _alpha2 == alpha2:
            return _alpha3
    else:
        return None


_records_list = []


for alpha2, name in data.LANGUAGES_CODE_TO_NAME_EN.items():
    string = f'Language("{name}", "{alpha2}", "{alpha3_by_alpha2(alpha2)}", ' + f"'{in_other_languages(alpha2)}')"
    # ic(string)
    _records_list.append(string)

_records_list = str(_records_list)
_records_list = _records_list.replace("'Language(", "Language(")
_records_list = _records_list.replace(")', Language(", "), \nLanguage(")

with open('json_test2test.txt', 'w') as file:
    file.write(_records_list)

[ZhymabekRoman]

• Do you need any help with the code?

I have one question - how do we implement caching? Will we make a specialized decorator or will we manually put values in the class attribute? I've just thought about this solution again - I think it's better to manually fill in the cache, it gives more flexibility. Only I'm thinking of making a cache (dictionary) with a limited number of elements, something like the LRU algorithm, and make it as a weak reference, so that if the device memory is not enough, interpreter resets the cache

[Animenosekai]

I have one question - how do we implement caching? Will we make a specialized decorator or will we manually put values in the class attribute? I've just thought about this solution again - I think it's better to manually fill in the cache, it gives more flexibility. Only I'm thinking of making a cache (dictionary) with a limited number of elements, something like the LRU algorithm, and make it as a weak reference, so that if the device memory is not enough, interpreter resets the cache

Yea I think it's better if we add the caches manually since it gives more flexibility and we are sure of what's caches and what's not.

especially since it works from python 2.7+

I mean, if we can get the whole library to support Python 2.7+ that would be great. We'll need to look into the parts which are making it not work under Python 2.

It seems to me there is no point in doing this, if a person (like me in the screenshot) enters the wrong language name or language code.

I mean, if the person entered a valid language but there is no result to it (some languages are missing one or two translations if I remember correctly)

It is not necessary to use fuzzywuzzy

If we can avoid adding dependencies that's nice but if the Levenshtein distance is better maybe

so now we can safely combine the BaseLanguage and Language classes

I meant why we needed a BaseLanguage class and not just use the Language.

I faced a problems that caused the interpreter to refuse to run the file

Hmmm that's weird, I'll look into that

[ZhymabekRoman]

I just thought - maybe the best way to use Microsoft Translate instead of Bing? I understand that they use the same engine, but Microsoft Translate supports more functions

[Animenosekai]

Well if we find any endpoint from Microsoft Translate usable why not.

We could use Bing Translate as a fallback since it's the same engine