Merge Data Access API and Tableau API credentials

[biblicabeebli]

These credentials are virtually identical, it was a mistake to separate them in the first place.

• Tableau security wrapper may have different well-defined ~400 class error codes so that needs to be retained.
• • will need to check with @hydawo on literally how those appear inside Tableau, it was developed blind 🙄.
• The engineering effort should be to merge the database entities and usage thereof into one table.
• Extend the multiple tableau key management to the new Everything keys.
• Add enhanced logging of keys? display a last use?
• Purge every reference to Tableau from everything other than the WDC and the specific wrappers.

(Where did the original issue for this go? I remember making it.)

[biblicabeebli]

Update - I did this yesterday.
The easiest way to do this was:

• ApiKey and the data access creds were (intentionally updated by me a while ago!!) designed to be literally identical at the password authentication level, so
• make a migration that takes the data access api key and create a new former-data-access-api-values instance of an api key
• remove old api key code
• replace all the resultant broken code with calls into ApiKey for the same functionality.
• fix fix fix all the broken tests that needed to reference updated database Things.
• and then it just worked. it was amazing.

I've updated the manage credentials page accordingly, removed former access api management endpoints.

I'm currently doing cleanup and The Great Tableau Terminology Purge (some stuff still is referred to as tableau credentials).

[biblicabeebli]

Last week saw some cleanup, this is effectively finished, but we need to update documentation (I believe there are screenshots to be updated).

[hydawo]

@biblicabeebli since this is live on staging only, let's keep it there for now.

At our next RA meeting, I'll have an RA update screenshots/instructions in our various manuals.

IMPORTANT: we also need to update our Forest scripts keyring.py file which asks for both sets of credentials. I'm unsure the implications to that code with the merging of these credentials...

TLDR: do not launch on other servers besides staging until I confirm we are ready. Moving this to Hassan's sprint

[biblicabeebli]

(This work and current state on staging is not considered production-ready.)

Old credentials will (should, if not that is a bug so please report it) continue to work, your old credentials will all still work.

We can also update the Keyring interface thing that Mano uses

[biblicabeebli]

This change seems to be working fine under all known circumstances, the remaining work is updated integration with Mano.

[biblicabeebli]

The tableau endpoint has been replicated (shared code and taking the exact same extensive filtering options) as a data api endpoint. The only difference between ...

• the Tableau endpoint: api/v0/studies/<str:study_object_id>/summary-statistics/daily
  and
• The data API endpoint: get-summary-statistics/v1
  is that the Tableau expects the study parameter to be named study_object_id, and that the id and secret keys to be named X-Access-Key-Id and X-Access-Key-Secret.
  The data api endpoint expects the study parameter be named study_id and the credentials be named access_key and secret_key

I don't see this documented any where, so here is some official documentation for the endpoint (both of them):

The list of extra parameters available to both versions of this endpoint are:

• end_date - formatted as YYYY-MM-DD.
• start_date - formatted as YYYY-MM-DD.
• limit - an integer number of results to return.
• ordered_by - any of the fields listed below.
• order_direction - a string "ascending" or "descending" (defaults to ascending, e.g. the lowest value will be first)
• participant_ids - a comma separated list of participant ids to limit the query to.
• fields - restrict the query to any of the fields listed below.

The following is a comprehensive (subject to change as updates are made for forest and the Beiwe Backend) list of all fields available for response data filtering and sorting:

# study_id is required detail of your query, and though the endpoint may also accept it as a sorting field, such sorting behavior is undefined.
# timezone is not recommended as a sorting field

# Basic fields
date
study_id
participant_id
timezone

# data quantity metrics
beiwe_accelerometer_bytes
beiwe_ambient_audio_bytes
beiwe_app_log_bytes
beiwe_bluetooth_bytes
beiwe_calls_bytes
beiwe_devicemotion_bytes
beiwe_gps_bytes
beiwe_gyro_bytes
beiwe_identifiers_bytes
beiwe_ios_log_bytes
beiwe_magnetometer_bytes
beiwe_power_state_bytes
beiwe_proximity_bytes
beiwe_reachability_bytes
beiwe_survey_answers_bytes
beiwe_survey_timings_bytes
beiwe_texts_bytes
beiwe_audio_recordings_bytes
beiwe_wifi_bytes

# Forest Jasmine Tree output
jasmine_distance_diameter
jasmine_distance_from_home
jasmine_distance_traveled
jasmine_flight_distance_average
jasmine_flight_distance_stddev
jasmine_flight_duration_average
jasmine_flight_duration_stddev
jasmine_gps_data_missing_duration
jasmine_home_duration
jasmine_gyration_radius
jasmine_significant_location_count
jasmine_significant_location_entropy
jasmine_pause_time
jasmine_obs_duration
jasmine_obs_day
jasmine_obs_night
jasmine_total_flight_time
jasmine_av_pause_duration
jasmine_sd_pause_duration

# Forest Willow Tree output
willow_incoming_text_count
willow_incoming_text_degree
willow_incoming_text_length
willow_outgoing_text_count
willow_outgoing_text_degree
willow_outgoing_text_length
willow_incoming_text_reciprocity
willow_outgoing_text_reciprocity
willow_outgoing_MMS_count
willow_incoming_MMS_count
willow_incoming_call_count
willow_incoming_call_degree
willow_incoming_call_duration
willow_outgoing_call_count
willow_outgoing_call_degree
willow_outgoing_call_duration
willow_missed_call_count
willow_missed_callers
willow_uniq_individual_call_or_text_count

# Forest Sycamore Tree output
sycamore_total_surveys
sycamore_total_completed_surveys
sycamore_total_opened_surveys
sycamore_average_time_to_submit
sycamore_average_time_to_open
sycamore_average_duration

# Forest Oak Tree output
oak_walking_time
oak_steps
oak_cadence

[biblicabeebli]

@hydawo you might want the above comment