Dataset search and mpeg dash support

FEATURES:
MET-1420 Improve saving xslt functionality
MET-1881 Impose list length limit for execution lists
MET-2360 Upgrade morphia version
MET-1480 Improve inqueue cancelling functionality
MET-2342 Provide support for Mpeg dash in media library
MET-2343 Add facet value for mpeg dash
MET-2311 Introduce dataset search
MET-2371: Indexing support for skos:inScheme within Concepts.
General technical debt improvements

BUGS:
MET-2347 Fix pending status and add SocketTimeoutException as cause and more clarity logs in case an exception occurs
MET-2377 Provide more flexibility in handling incorrect mime types
MET-2478 Fix raw xml executions list retrieval
MET-2370 Provide more thorough tika management to detect types

Restrict executions to the latest harvest for a dataset, fix blocking issues on media and dereference

FEATURES:
MET-2297: Add support for edm codecName for audio resources
MET-2261: Extract codecName for audio resources
MET-2278, MET-2193: Technical debt improvements
MET-2260: Extend library methods for facets
MET-2245: Restrict executions to latest harvest
MET-2257: Change gray to grayscale in web resource information in mongo
MET-2235: Create light api requests for plugin selection
MET-2194: Integrate ecloud changes regarding multiple files/representations retrievals

BUGS:
MET-2274: Introduce timeouts for tika connections
MET-2281: Fix null pointer on dataset execution information
MET-2272: Be permissive during dereferencing for internal links
MET-2219: Make thumbnail mime type available
Fix validation schema location error
Add explicit import of BadRequest

Timestamp dates fixes, introduction of DELETED tag

FEATURES:
MET-2107: Circular dependencies resolving
MET-1525: Remove datasetId field from specific plugins and provide in general on creation of a task
MET-1936: Unit tests for tier calculation
MET-2169: Guard canceling user state
MET-2153: Also save disabled workflow plugins, with ui having more control on the workflows now
MET-2114: Workflow validation improvements
MET-2026: Remove media service bypass. Refactoring ExecutionRules
MET-2131: Support decoding fake facets
MET-1696: Introduce the DELETED tag for plugins.
MET-1897: Update zoho getOrganizations according to requirements

BUGS:
MET-2182: Creation and Update date fix
MET-2220: If result from the enrichment comes with a non matching resourceId, then dereference the provided resourceId
MET-2199: Fix the logic of computing the edm:preview
MET-2171: Last preview and publish information takes into account also reindexing operations
MET-2172: File size should not be negative after media processing
MET-2130: Refactoring Workflow validation - linkchecking rules
MET-2006: Download timeout for resource downloading in indexing library.
MET-2151: Fix build: force travis to use right distribution for java 8.
MET-2115: Reduce logs during build for travis
MET-2111: Mime type analysis fixes
MET-2031: Reduced media processing isShownAt
Fix jacoco coverage for sonar
Fix jacoco coverage for travis
Fix validation schema location error
Prevent double occurrence of pluginType in plugin and metadata.

Support for tier calculation annotations

FEATURES:
MET-1928: Tier calculation during indexing
MET-1988: Tier calculation improvements
MET-2008: Tier annotations serialization during indexing
MET-1983: make resource download connect/socket timeouts configurable
MET-1867: Upgrade mongo, redisson library versions
MET-1982: Processor pool for media extractors and link processors
MET-1833: Upgrade postgresql, hibernate, rabbitmq libraries
MET-2009: URL of source link added in Link Checking library exceptions
MET-2021: Link checking step can also occur after any other plugin
MET-1985: Update schema files
MET-2050: Attempt to solve some high rated vulnerabilities for technical debt
MET-2032: Change dereferenceable fields
MET-2042: Default Xslt is automatically inserted in db for new installations
Better error message in media processing for audio/video files
Upgrade xstream, jackson library version

BUGS:
MET-1676: Add missing dependencies to media service code
MET-1949: Media processor bug resolving and optimization
MET-1952: Catch any potential throwable inside consumer to ensure ack is sent
MET-1981: Fix erroneous way of getting owner element of attribute
MET-2002: Be more flexible for null list for objects in metadata cache
MET-2030: Stop considering isShownAt resources as full text for Solr
MET-2007: Fake facets and has_media only for isShownBy and hasView
MET-2102, MET-2099, MET-2045: Fixes for media tiers
MET-2110: Update indexing to accept application/xhtml as text
Small fix to prevent null pointer exception in reprocessing script

Preparation for data cleaning and reprocessing

Relative tickets:

FEATURE: MET-1948_no_zero_facet_codes_generated
FEATURE: MET-1894_allow_filtering_on_all_executions
FEATURE: MET-1625_Normalization_of_xml_lang_attributes
FEATURE: MET-1762_remove_mocked_periodic_loop_from_executor
FEATURE: MET-1811_dashboard_executions_improvements
FEATURE: MET-1600_cleanup_dependencies_for_logging
FEATURE: MET-1579_update_code_communication_with_zoho
FEATURE: MET-1826_Backend_support_for_new_executions_list
FEATURE: MET-1624_add_normalized_language_in_Europeana_proxy
FEATURE: MET-1826_Backend_support_for_new_executions_list
FEATURE: MET-1889_register_reindexed_and_invalidated_plugins

BUG: MET-1956_language_normalization_preserve_subtags
BUG: MET-1955_handle_absence_of_content_type
BUG: MET-1924_timeout_on_ImageMagick
BUG: MET-1860_Thumbnail_image_type_not_correct

Link Checking Refactoring

Relative tickets:

FEATURE : MET-1763_support_for_sending_previous_task_identifier
FEATURE : MET-1806_Add_call_for_additional_node_statistics
FEATURE : MET-1770_send_sampling_parameter_to_ecloud
FEATURE : MET-1799_bypass_workflows_for_dps_gap' into develop
FEATURE : MET-1781_limit_timespan_entities_up_to_century_urls
FEATURE : MET-1764_indefinite_consecutive_monitor_cancel_failures
FEATURE : MET-1786_Create_API_for_record_history_viewing
FEATURE : MET-1783_add_main_wikidata_resource_to_same_as
FEATURE : MET-1741_disable_preview_collections_buttons_at_cleaning
FEATURE : MET-1766_proxy_call_for_checking_if_report_exists
FEATURE : MET-1768_add_sampling_parameter_for_link_checking
FEATURE : MET-1751_link_checking_always_allowed
FEATURE : MET-1750_Link_checking_support_for_EDM_external
FEATURE : MET-1761_some_API_changes_link_checking
FEATURE : MET-1723_send_cancellation_reason_to_ecloud
FEATURE : EA-1468_EA-1471_organizations_foaf_depiction_and_address_has_geo
FEATURE : MET-1678_error_report_for_failed_task
FEATURE : MET-1712_support_more_information_when_a_cancellation_occurs
FEATURE : MET-1719_send_parameters_through_body_of_put_method
FEATURE : MET-1533_Improve_color_space_support

BUG : MET-1779_elements_that_have_entities_should_be_part_of_europeana_proxy_with_ref_to_the_entity
BUG : MET-1675_fix_media_servers_for_FLV_files
BUG : MET-1743_validation_file_system_collisions

DEBT : MET-1638_unit_tests_for_media_service
DEBT : MET-1211_indexing_library_technical_debt
DEBT : MET-1797_Unit_tests_for_record_history_code
DEBT : MET-1821_clearup_testobjectfactory

This release is focused on the re-implementation of the link checking library.

Media Service refactoring

Relative tickets:

• MET-1563 Create API for topology using old code
• MET-1578 magick invocation: wrong command arguments
• MET-1565 Refactor code behind API
• MET-1604 MMS: misidentification of WR content type
• MET-1637 Incorrect isPartOf value for Timespans in Mongo
• MET-1595 Newlines stripped from dc:description

This release is focused on the re-implementation of the media service library.
dc:description was changed to not strip newlines during normalization.
Wrong value on timespans in mongo was fixed/updated.

Dereference/Enrichment/Media process fixes

Relative tickets:

• MET-1576 Fix Enrichment/Dereference issues
• MET-1593 Fix recognition of "grayscale" images in media library

Detailed changes:

• Proxy entity fields hasPart and isPartOf are not dereferenced anymore
• WebResource entity the hasPartList is not dereferenced anymore
• Added retries for "HttpServerErrorException" type exceptions on dereference and enrichment external calls
• Fixed the check from "grayscale" to checking "Gray" value during the setColorspace in WebResource of the
  media library

First release of Metis

With this release we aim to have an improved working solution, over the previous system, of a full workflow cycle for our ingestion pipeline, including all the new features and fixes that Metis had in its design.