Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IMPROVE ENRICHMENT EXTRACTION INTERFACE: As Dan, I want the Enrichment Extraction interface updated, so that the workflow is intuitive and I can better customise my extraction #45

Merged
merged 35 commits into from
Oct 24, 2023

Conversation

richardmatthewsdev
Copy link
Contributor

@richardmatthewsdev richardmatthewsdev commented Oct 16, 2023

Acceptance Criteria

  • The user is easily able to navigate the process and understand the 2 concepts involved (API record vs Enrichment URL)
  • Preview presents response in a useful way
  • Preview is helpful with errors and isn't a dead end if the first record doesn't meet requirements.
  • Conditions allow enrichment to skip based on record metadata (eg skip category:Audio)
  • Target (destination) where enrichment is pulling from is intuitive (and automatic where possible)

Details

  • API record selection
  • source_id - could be auto-populated from pipeline (but might need to deal with being empty?)
  • Conditions - allow enrichment to skip based on record metadata (eg skip category:Audio)
  • Enrichment details
  • URL construction (involving base_url + record metadata)
  • Format (html, XML etc)
  • Headers (eg optional authentication)
  • Throttle - only applied to external requests (with a valid URL)
  • HTTP verb (get/post)

Risks

  • What problems could this cause (mitigations? notify dev-all-dnz?)
  • Is there some extra investment that could add value/learning/QA

Notes

  • This Enrichment needs Auth details in a header
  • The transformation error feedback about the "wrong type" would be really useful for when this goes wrong record['dc_identifier'] and needs a .first.to_s. Or when Dan uses a symbol instead of a string.
  • Also the Preview functionality that shows the actual URL being called will be super useful.
  • BUG: The enrichment URL evaluation where it checks the curly brackets is brittle. Things like "#{record['hello']}/iamcool/#{record['goodbye']}" will likely not work. Also if the URL is just record['landing_url'] (without quotes or curly brackets) it will fail

Notes

Gus has said that we need the ability to enrich from an API source, this could be a POST request with some headers and slugs based on information in the record.
Harvest From PDF

enrichment_extraction_improvements.mov

create an Enrichment Extraction by a modal, following the same pattern as a harvest extraction.
@richardmatthewsdev richardmatthewsdev changed the base branch from main to rm/workflow-enhancements-1 October 16, 2023 00:29
Add settings modal to the Enrichment Extraction view.
Initial stab at getting the Enrichment to work with the request and parameters.
Improve the way we extract records from the API response to be used in the extraction.
Remove the source_id and the test button as they can be inferred from the pipeline and dont need to be tested.
Scope the API record on the enrichment extraction preview modal to just one record.
Base automatically changed from rm/workflow-enhancements-1 to main October 18, 2023 19:06
Next and previous through Extraction Preview
Allow the user to next and previous through the Enrichment response
Fix issue where deleting an enrichment would mess up the following priorities.
Use the destination from the pipeline job if the enrichment is run as part of a pipeline.
Fix alignment of information in Enrichment Preview Modal.
Fix width of slug value on query builder.
Make sure that the URL when quering for enrichments is valid.
Improve Active Parameter settings when interacting with the Parmeter List Panel
Add Active Fields to the Transformation App.
Improve feedback when previewing the record selector on the Transformation Definition.
Base the fallback record selector on the format for more consistent results.
Cover error state for a evaluated parameter in tests.
Create transformation of appropriate kind when coming from the extraction preview
Copy link
Contributor

@motizuki motizuki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice improvement!

Copy link

@lunmanlee lunmanlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Win win @richardmatthewsdev !! Thanks for including the video 💇‍♀️

The API record format will always be JSON in the preview window.
@richardmatthewsdev richardmatthewsdev merged commit bd2c1b0 into main Oct 24, 2023
7 checks passed
@richardmatthewsdev richardmatthewsdev deleted the rm/improve-enrichment-interface branch October 24, 2023 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants