Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve seeder #466

Merged
merged 5 commits into from
Nov 23, 2024
Merged

Improve seeder #466

merged 5 commits into from
Nov 23, 2024

Conversation

fityannugroho
Copy link
Owner

@fityannugroho fityannugroho commented Nov 23, 2024

PR Checklist

Please check if your PR fulfills the following requirements:

Put [x] to check

  • The commit message follows our Contributing Guidelines
  • Tests for the changes have been added (optional, for bug fixes or features)
  • Docs have been added / updated (optional, for bug fixes or features)

PR Type

What kind of change does this PR introduce?

Please check any kind of changes that applies to this PR using [x]

  • Bug fix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes

What is the current behavior?

Please describe the current behavior that you are modifying, or link to a relevant issue.

Issue Number: N/A

Seeder is slow and inefficient because Seeder.hasDataChanges retrieve all data from database and compare it with the data from package.

What is the new behavior?

This pull request includes significant changes to the seeder logic and database schema across multiple database systems. The most important changes include the addition of a new SeederLogs model, refactoring of the seeder methods to simplify the code, and the introduction of a utility function to get the installed package version.

Now hasDataChanged is determined by the version of installed idn-area-data package and the latest version of saved data in database (in the SeederLogs).

Database schema updates:

  • Added SeederLogs model to the schemas for MongoDB, MySQL, PostgreSQL, and SQLite. This model includes fields for id, dataVersion, and createdAt to track seeder execution logs. [1] [2] [3] [4]

Seeder refactoring:

  • Refactored the MongodbSeeder class to remove redundant methods for checking data changes and deleting collections. Introduced a generic deleteAreas method to handle deletion for all area types.
  • Updated the main seeder script to use the new deleteAreas and insertAreas methods, streamlining the deletion and insertion processes.

Utility function:

  • Added a new utility function getInstalledPackageVersion to obtain the version of an installed package using pnpm list --json. This function is used in the seeder to compare the installed package version with the latest data version in the database.

Testing:

  • Introduced tests for the getInstalledPackageVersion function to verify its behavior under different scenarios, such as when the package is found, not found, or an error occurs.

Other information

You need to run db:migrate and db:seed to apply this changes.

@fityannugroho fityannugroho added the enhancement New feature or request label Nov 23, 2024
@fityannugroho fityannugroho merged commit dcf7ef4 into main Nov 23, 2024
11 checks passed
@fityannugroho fityannugroho deleted the improve-seeder branch November 23, 2024 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant