Using a component inside a loop will duplicate the database query (or rendering the entire page) for each element

[medycynka]

Hi!

While trying to use this wonderful library in our project, we encountered a performance issue. A table that contains only 80 rows renders about 12 seconds through this piece of code:

{% for obj in filtered_object_list %}
    {% unicorn 'available_projects_row' counter=forloop.counter columns_choice=columns_choice key=forloop.counter obj=obj %}
 {% endfor %}

According to django-debug-toolbar:

If instead of using the component {% unicorn ... %} we generate the table in the "classic" way using pure html, the page load time is less than a second (with around 90 database queries). What we have already tried:

• Changing djang models to simple objects or dictionaries before page rendering
• Disabling caching of components by overriding the methods of the UnicornView class.

Therefore, we guess that the problem may be the way individual components are rendered, but we have not been able to find the cause or solution to the problem so far.

Due to the fact that this is a project with no access to the source code, I am not able to share the full data of the data but only an illustrative one:

class AvailableProjectsRowView(UnicornView):
    template_name = 'unicorn/available_projects_row.html'
    project_versions: list = []
    project_process_statuses: dict = None
    obj: AvailableProjectPresenter = None
    counter: int = 0
    columns_choice: dict = None
    loaded_status_table: bool = False
    selected_columns: int = 0
    response_class = UnicornTemplateResponse

    def mount(self):
        self.project_versions = []
        self.project_process_statuses = {}
        self.counter = self.component_kwargs.get('counter', 1)
        self.loaded_status_table = False
        self.obj = self.component_kwargs.get('obj')
        self.columns_choice = self.component_kwargs.get('columns_choice')
        self.selected_columns = sum(column.checked or column.disabled for column in self.columns_choice.values())


...


class Project(models.Model):
    institution = models.ForeignKey(...)
    name = models.TextField(...)
    short_name = models.CharField(...)
    description = models.TextField(...)
    undertaking_type = models.CharField(...)
    sor_name = models.CharField(...)
    sor_type = models.CharField(...)
    sor_period = models.CharField(...)
    scope = models.CharField(...)
    project_type = models.CharField(...)
    category = models.CharField(...)
    signature = models.CharField(...)
    contract_no = models.CharField(...)
    current_phase = models.CharField(...)
    # more model fields, but their number did not affect performance because we also tested converting model instances to 
    # simple objects or dictionaries

...


class AvailableProjectPresenter:
    def __init__(self, project, request_user, selectable):
        self.project = project
        self.enterprise_name = project._meta.model_name
        self.request_user = request_user
        self.show_full_data = selectable
        self.is_selectable = selectable

    @property
    def name(self):
        return self.project.short_name if self.project.name else ''

    @property
    def short_name(self):
        return self.project.short_name

    @property
    def institution(self):
        return self.project.institution.abbreviation
        
    # more properties

Currently we are stuck and do not know what we can do next, and we want to integrate our project with this library because it will improve our work and provide better code clarity.

[adamghill]

Hello, I'm sorry you are running into this issue. Off the top of my head, I wonder if this is an n+1 problem? You don't show the query for filtered_object_list, but you might be able to put a select_related or prefetch_related which would reduce subsequent foreign-key lookups. Although based on your statement that you have tried "changing Django models to simple objects" this might not solve your problem.

One thing to see what might be going on is to change the Unicorn component to a regular Django include and output the same model data. If it's still slow, it's an ORM issue. If it's super fast, then there might be something going on in Unicorn.

I render a component for every row in a table in this example: https://github.com/adamghill/django-unicorn/blob/main/example/unicorn/templates/unicorn/nested/table.html#L38. The backend code is here: https://github.com/adamghill/django-unicorn/blob/main/example/unicorn/components/nested/table.py#L31. You can see I have a select_related to prevent duplicate database queries. I just tested it with 100 rows and it still renders very quickly, so hopefully tweaking the ORM query will work for you as well!

There should be minimal overhead when rendering components with Unicorn, but If the above doesn't work, please let me know.

[medycynka]

Hi,
Thank you very much for your quick reply. Forgive me, I forgot to include that filtered_object_list is basically a converted QuerySet of projects to a list of objects of type "AvailableProjectPresenter" in a way:

context['filtered_object_list'] = [
    AvailableProjectPresenter(project, ...) for project in Project.objects.select_related(...).prefetch_related(...).filter(...).annotate(...)
]

Unfortunately, we are sure that this is not an N+1 problem because:

• we are already using select_related and prefetch_related and we use it on the correct fields
• we tested this problem on simplified queries without using any related models
• in the template we only changed {% include ... %} to {% unicorn ... %} and this increased the number of queries from 55 to 685 (of which almost 95% were duplicates) and the wait time increased from ~800ms to ~11s
• according to django-debug-toolbar every database query called in our template is duplicated, not only the Project.objects... query, but also other queries used in the template

We suspect it may be a problem with the way components render themselves and the use of beautifulsoup4 to find a component's parent, but we haven't been able to test that yet.

[adamghill]

Ah, thanks for the further clarification here -- that is frustrating! I'm going to try to replicate your set up in the example app in this repo to see if I get the same results. Or let me know if there is a minimal way to reproduce, thanks.