FileTypeRouter Improvement

[jlonge4]

I have noticed that in a linux env, for instance an aws lambda (python3.12), the FileTypeRouter will output docx and pptx (or other microsoft based flavors of files) as unclassified unless you first run:

import mimetypes
# Add .docx MIME type
mimetypes.add_type("application/vnd.openxmlformats-officedocument.wordprocessingml.document", ".docx")

How could we implement checking/adding mime types specified at init time are added at init time, reducing unclassified outputs on legitimate mime types.

Any ideas @anakin87 ?

[anakin87]

👋

On Ubuntu 22.04, python 3.8, this works:

from haystack.components.routers import FileTypeRouter

file_type_router = FileTypeRouter(mime_types=["text/plain", "application/pdf", "text/markdown", "application/vnd.openxmlformats-officedocument.wordprocessingml.document"])

path = "MYFILE.docx"

print(file_type_router.run([path]))

>>> {'application/vnd.openxmlformats-officedocument.wordprocessingml.document': [PosixPath('MYFILE.docx')]}

Can you try it in your system? Am I missing something?

[jlonge4]

@anakin87 Ubuntu most likely has it due to the libre office packages I would imagine, however inside a python lambda function:

import json
import mimetypes

def lambda_handler(event, context):
    guess = mimetypes.guess_type('application/vnd.openxmlformats-officedocument.wordprocessingml.document')
    return {
        'statusCode': 200,
        'body': json.dumps(str(guess))
    }

>>> {"statusCode": 200, "body": "\"(None, None)\""}

[anakin87]

Ah...
@vblagoje you worked on this if I remember correctly.
Do you have any ideas/suggestions?

[jlonge4]

@anakin87 @vblagoje

Just for good measure, I added a layer and tried your exact code:

{
  "statusCode": 200,
  "body": "\"{'unclassified': [PosixPath('MYFILE.docx')]}\""
}

[vblagoje]

Hey @jlonge4 , @anakin87 and I spoke about this and adding some additional_mimetypes: Dict[str, str] parameter would make the most sense as there are no guarantees how defaults mappings are resolved across various runtime environments and Python versions I suppose.

Would you mind opening a PR for this @jlonge4 ? We'll review and integrate it soon after

[jlonge4]

@vblagoje @anakin87 Absolutely, will have it ready shortly!