Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIMSBIOHUB-633: Revamp CSV Imports - Critter #1438

Merged
merged 42 commits into from
Dec 7, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
fe045e6
chore: partially done structure for new csv import
MacQSL Oct 28, 2024
8c0cc12
feat: validate method
MacQSL Oct 30, 2024
8062962
chore: wip
MacQSL Nov 19, 2024
a999b2d
feat: progress...
MacQSL Nov 20, 2024
79f488f
feat: added validators for cell values
MacQSL Nov 20, 2024
853cfe1
chore: moved files
MacQSL Nov 20, 2024
a0c4f87
feat: updated docs for types
MacQSL Nov 21, 2024
cb0a7b9
feat: created new critter strategy
MacQSL Nov 21, 2024
73a1f50
chore: still working on strucutre for config
MacQSL Nov 21, 2024
1cd7235
feat: updated helper functions for csv-utils
MacQSL Nov 25, 2024
c5d9b71
feat: updated import csv critters
MacQSL Nov 26, 2024
0dffc39
feat: added new config header util files
MacQSL Nov 27, 2024
834934f
feat: added files for csv-header-config validation / setter functions
MacQSL Nov 27, 2024
08709c1
chore: updated tests for cell validators
MacQSL Nov 28, 2024
4e742e6
feat: updated some structure + moved validation logic into class
MacQSL Nov 28, 2024
110766d
feat: moved some redundant params around + seperated CSVHeaderConfig …
MacQSL Nov 29, 2024
2b0469b
fix: updated some functions in csv-config-validation
MacQSL Nov 29, 2024
29c51b6
chore: tests for rorEachCSVCell
MacQSL Nov 29, 2024
2427513
feat: updated error formatting to be in the parent
MacQSL Nov 29, 2024
e1e563f
feat: partial PR comments
MacQSL Nov 29, 2024
61aad38
feat: PR comments + additional JSDocs
MacQSL Nov 29, 2024
410d0f4
feat: created type for cell setter and validator
MacQSL Nov 29, 2024
eb2b8aa
feat: updated tests for CSVConfigUtils
MacQSL Nov 29, 2024
7807f4e
feat: updated types in CSVConfigUtils
MacQSL Nov 29, 2024
75374ca
feat: updated getters in CSVConfigUtils
MacQSL Nov 30, 2024
2e230fa
chore: resolving broken tests
MacQSL Nov 30, 2024
d0494f4
feat: updated tests
MacQSL Nov 30, 2024
1ae762d
feat: added nested record class
MacQSL Dec 2, 2024
7cc6f4a
feat: added tests for endpoint
MacQSL Dec 3, 2024
32b4c89
fix: pr comments
MacQSL Dec 3, 2024
52f5c46
Merge branch 'dev' into mdColumnValidator
MacQSL Dec 4, 2024
ace5d05
fix: updated tests + included unique check for wlh_id
MacQSL Dec 4, 2024
e3b3435
Merge branch 'dev' into mdColumnValidator
MacQSL Dec 4, 2024
4b55647
feat: added errors to error dialog in frontend
MacQSL Dec 4, 2024
e7cd180
Merge branch 'mdColumnValidator' of https://github.com/bcgov/biohubbc…
MacQSL Dec 4, 2024
c0e2d8b
update solutions text for import errors
mauberti-bc Dec 6, 2024
d4f0403
fix: adding optional flag for header
MacQSL Dec 6, 2024
99996e0
Merge branch 'mdColumnValidator' of https://github.com/bcgov/biohubbc…
MacQSL Dec 6, 2024
6b52461
fix: sex now optional
MacQSL Dec 6, 2024
c874004
fix: sex now optional
MacQSL Dec 6, 2024
1fc0b64
Merge branch 'mdColumnValidator' of https://github.com/bcgov/biohubbc…
MacQSL Dec 6, 2024
77bac9c
Merge branch 'dev' into mdColumnValidator
MacQSL Dec 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
import { getDBConnection } from '../../../../../../database/db';
import { csvFileSchema } from '../../../../../../openapi/schemas/file';
import { authorizeRequestHandler } from '../../../../../../request-handlers/security/authorization';
import { ImportCrittersStrategy } from '../../../../../../services/import-services/critter/import-critters-strategy';
import { importCSV } from '../../../../../../services/import-services/import-csv';
import { ImportCrittersService } from '../../../../../../services/import-services/critter/import-critters-service';
import { getLogger } from '../../../../../../utils/logger';
import { parseMulterFile } from '../../../../../../utils/media/media-utils';
import { getFileFromRequest } from '../../../../../../utils/request';
import { constructXLSXWorkbook, getDefaultWorksheet } from '../../../../../../utils/xlsx-utils/worksheet-utils';

const defaultLog = getLogger('/api/project/{projectId}/survey/{surveyId}/critters/import');

Expand Down Expand Up @@ -83,29 +83,56 @@
},
responses: {
200: {
description: 'Import OK',
description: 'Import OK'
},
400: {
$ref: '#/components/responses/400'
},
422: {
description: 'CSV validation errors',
content: {
'application/json': {
schema: {
type: 'object',
additionalProperties: false,
required: ['survey_critter_ids'],
required: ['validation_errors'],
properties: {
survey_critter_ids: {
validation_errors: {
type: 'array',
items: {
type: 'integer',
minimum: 1
type: 'object',
additionalProperties: false,
required: ['error', 'solution', 'rowIndex', 'header'],
properties: {
error: {
type: 'string'
},
solution: {
type: 'string'
},
cell: {
oneOf: [{ type: 'string' }, { type: 'number' }]
},
header: {
type: 'string'
},
rowIndex: {
type: 'number'
},
values: {
type: 'array',
items: {
oneOf: [{ type: 'string' }, { type: 'number' }]
}
}
}
}
}
}
}
}
}
},
400: {
$ref: '#/components/responses/400'
},
401: {
$ref: '#/components/responses/401'
},
Expand Down Expand Up @@ -133,19 +160,23 @@

const connection = getDBConnection(req.keycloak_token);

const mediaFile = parseMulterFile(rawFile);
const worksheet = getDefaultWorksheet(constructXLSXWorkbook(mediaFile));

Check warning on line 164 in api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts

View check run for this annotation

Codecov / codecov/patch

api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts#L163-L164

Added lines #L163 - L164 were not covered by tests

try {
await connection.open();

// Critter CSV import strategy - child of CSVImportStrategy
const importCsvCritters = new ImportCrittersStrategy(connection, surveyId);
const importService = new ImportCrittersService(connection, worksheet, surveyId);

Check warning on line 169 in api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts

View check run for this annotation

Codecov / codecov/patch

api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts#L169

Added line #L169 was not covered by tests

const surveyCritterIds = await importCSV(parseMulterFile(rawFile), importCsvCritters);
const errors = await importService.importCSVWorksheet();

Check warning on line 171 in api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts

View check run for this annotation

Codecov / codecov/patch

api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts#L171

Added line #L171 was not covered by tests

defaultLog.info({ label: 'importCritterCsv', message: 'result', survey_critter_ids: surveyCritterIds });
if (errors.length) {
return res.status(422).json({ validation_errors: errors });

Check warning on line 174 in api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts

View check run for this annotation

Codecov / codecov/patch

api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts#L174

Added line #L174 was not covered by tests
}

await connection.commit();

return res.status(200).json({ survey_critter_ids: surveyCritterIds });
return res.status(200).send();

Check warning on line 179 in api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts

View check run for this annotation

Codecov / codecov/patch

api/src/paths/project/{projectId}/survey/{surveyId}/critters/import.ts#L179

Added line #L179 was not covered by tests
} catch (error) {
defaultLog.error({ label: 'importCritterCsv', message: 'error', error });
await connection.rollback();
Expand Down
Loading
Loading