Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add time graph for annotator contributions #8

Open
xavierfav opened this issue Mar 29, 2017 · 4 comments
Open

Add time graph for annotator contributions #8

xavierfav opened this issue Mar 29, 2017 · 4 comments

Comments

@xavierfav
Copy link
Contributor

It would be nice to have a graph to visualize annotators contributions across time in the contribute page with "Annotators' ranking".
Not so useful, but could maybe motivate people in their progression...

@ffont
Copy link
Member

ffont commented Mar 29, 2017

I think this is in fact useful. The ranking is computed in the compute_annotators_ranking function. For every user, the total number of contributions is the sum of Annotation and Vote objects under her name (n_annotations = Annotation.objects.filter(created_by=user, sound_dataset__dataset=dataset).count() and n_votes = Vote.objects.filter(created_by=user, annotation__sound_dataset__dataset=dataset).count()).

To plot a graph showing, you should filter the above queries by the created_at property so you can get "total number of annotations by user X before date Y" and "total number of votes by user X before date Y" (to dot hat you need to understand how the Django ORM works, see this related example). If you can do that then you only need to define a range of dates (i.e. last 10 hours) and run the queries for each user and for each date. With that you get the data to build your "evolution" plot for the last 10 hours.

To actually plot it you can use any javascript graph plotting library. In Audio Commons mediator we use CanvasJS. See that example: http://canvasjs.com/editor/?id=http://canvasjs.com/example/gallery/line/site_traffic/ is quite easy.

ffont added a commit that referenced this issue Apr 5, 2017
@ffont
Copy link
Member

ffont commented May 18, 2017

I came across this project: https://github.com/wa0x6e/cal-heatmap and I thought that maybe we could use something like this in Freesound datasets to see recent contributions to a dataset.

@jordipons
Copy link
Contributor

I also think these visualizations are useful in our case - IMO these can be seen as a gamification.

But just highlighting number of contributions, might be dangerous since some workers focus on going "fast" rather than getting "good" annotations. In order to acknowledge "good" workes, I propose including some sort of quality weighting to these curves which can be based on the quality estimate discussed in #31.

To this end, we could either weight the proposed curve, or show two rankings: i) # of contributions or ii) quality.

@xavierfav
Copy link
Contributor Author

It is a good idea to use quality to weight the contribution graph.
One simple method done in crowdsourcing for calculating the reward, is to count the votes that are considered right (several worker voting with the same answer).
In our case we want two workers to give the same vote.
Thus, this weighted contribution graph cannot be done in "real time".
But we could calculate the total "valid" contributions of users each month/week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants