Custom language detector node

[stazam]

Hi, I am building Haystack indexing pipeline, and as I understood from documentation PrePorcessors are language specific, while they work good with english, they are also pretty decent with different languages, but to improve perfomance I would need to specify language.

Question: So I would want to build CustomLanguageDetector, which would route documents into three different Preprocessors somehting like this?

from langdetect import detect
    
class CustomLanguageClassifier(BaseComponent):
        outgoing_edges = 3

        def run(self, document: Document):
            
            language = detect(document.content)
            if language == 'en':
                return {"documents":document}, "output_1"
            elif language == 'gr':
                return {"documents":document}, "output_2"               
            elif language == 'cz':
                return {"documents":document}, "output_3"
            else:
                raise ValueError('You are using docuemnt with unsupported language!')

        def run_batch(self, documents:List[Document]):
            pass

But I am getting an error lie this one:

TypeError: run() missing 1 required positional argument: 'document'

Did someone make the same thing and could help me solve the issue?
Thanks a lot inadvance.

[ZanSara]

Hello! Your approach is 90% correct! There are a couple of things to know.

1. run takes an argument called documents. Mind that it contains a list of documents, so in the body of the function you have to iterate over them.
2. run can produce output only on a single edge at a time. So if your documents list contains docs in different languages, you must discard some or throw an exception. In your case, this might be an issue...

There are other ways to approach this problem, so don't worry yet 😄 First of all I need to know, how does your pipeline look like, and how did you write your documents into the document store? What I have in mind is that you should rather add a language metadata tag to the documents before writing them into the document store. So when you need to retrieve them, you can filter them by language. I don't know if this approach is viable in your setup though, so let me know! I will help you along.

[ZanSara]

Oh wait, your repo is not a fork of Haystack? 😅 Could you move the changes on a fork, so we can open a PR?

To make a fork you should simply go on the Haystack repository and click on the Fork button ion the top-right. That will create a repository called haystack in your space, from which you can contribute changes to the original repository. Let me know if you need help with the process 😊

[stazam]

Hi, I made the fork in my repo here: https://github.com/stazam/LanguageClassifier (branch sza-language-classifier) and definitely, feel free to add any changes which made it compatible. :)

[stazam]

I am not sure, do u need some permission, which I should set up (sorry for complications, this is my first commmit to open source :D ).

[ZanSara]

Looks like it's all setup properly! I created a Pull Request for your changes here: #2994 (it was done by pressing the Contribute button in the top-right corner of the files browser, where the latest commit info is).

From now, all changes done on the sza-language-classifier branch of your repo will be visible in that PR and, once it's ready, we will review and merge it. If you find some improvement that can be added, please commit to that branch, so we will share them. Please do not delete the forked repo before the PR is merged, or the changes will be gone.

If you have more question you can add comments below the PR: they will be more visible for me. Thank you!

[ZanSara]

I need only one more action from you end: can you follow the instructions on this page to allow edits? https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests That's the only permission missing.