Cookie consent

AstroBin saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing AstroBin without changing the browser settings, you grant us permission to store that information on your device.

I agree

The time has come: new IOTD selection process (and call for moderators)

siovene
14 Oct, 2016 21:24
It's now been almost a year since the very long and infamous thread about improving the IOTD. I have been able to complete a lot of work on AstroBin that was more important than the IOTD, but now finally that time has come: the IOTD selection process will change.

The new process will be more community driven, so I will need some volunteers, but we'll get to that later.

What follows is a simple explanation on how the process will work.

There will be 3 groups involved in the process:

  1. Submitters (30 positions available)
  2. Reviewers (15 positions available)
  3. Judges (7 positions available)
Submitters will submit images to the Submission Queue during their normal browsing activities, via a dedicated new button.

Reviewers will express their vote on these submissions, advancing the best of them to the Review Queue.

Judges will actually elect some of the images in the Review Queue to "IOTD".

To ensure integrity and fairness, the following restrictions will be in place:

  • No one can submit, review or elect their own images (obviously)
  • If a user covers more than a role, they cannot advance an image that they have previously advanced
  • Judges cannot be Submitters too
  • After each yearly quarter, activities are reviewed and people are either asked to continue in their role or step down (initially I will take care of this, but I'm all up for forming a board to do it)
To ensure that all images get a chance, the moving window for the IOTD will be a lot larger than it is now (from one week, to potentially up to 8 weeks).

With this process, that was largely conceived by Paddy Gilliland, whom I thank profusely for his incessant suggestions and commentary  smile , we aim to achieve the following:

  • A more consistently high bar of excellency for the IOTD
  • A better chance for images that might have been missed or shadowed by others as things are at the time of writing
  • An involvement by a larger number of parties, to achieve greater consistency and quality
  • The removal of the "popularity context" factor, thanks to the elimination of the Likes from the equation
I believe that with the help of the community, these things can be achieved with this process.

I am now calling for volunteers to cover the Submitter and Reviewer roles. At this time, most  Judges have already been selected on a personal trust or recommendation basis.

Please note that being a Submitter or a Reviewer only requires 5-15 minutes of your time, and you don't have to do it every day. If we struggle, we can recruit more Submitters or Reviewers than the initial targets of 30 and 15 respectively.

To volunteer for a role, please simply reply to this thread with "I volunteer as a (submitter/reviewer)".
You can also use this thread to comment, criticise or offer suggestions for the new process.

However please know that this is now already implemented, and I'm just needing submitters/reviewers to actually push it to the server :-) I will of course do minor corrections based on your input but it's unlikely that I scrap the whole idea or change it substantially.

Thanks for reading!
Salvatore
Hondo
14 Oct, 2016 23:01
Even though you have removed the "likes" factor, I still think it will become a popularity contest due to the number of followers.  If the submitters and reviewers were not following anyone then that would improve the chances of a fair IOTD.  The judges should not be allowed to follow anyone either.  In reference to the infamous thread from a year ago, has anyone actually clicked the "opt out" button?  Based on my perusal of the AB membership and past IOTD's, I see that no one has selected it even though they were strong proponents of the opt out function at the time.  Are the submitters, reviewers, and judges eligible for the IOTD?  I hope not, otherwise you just defeated the whole purpose of trying to improve the IOTD.

Scott
qbool
15 Oct, 2016 00:20
Imho, there's a lot to like about this approach but Scott brings up a good point: how do Submitters become aware of images that should be evaluated for submission? Is the intent that they tirelessly and relentlessly browse and scan for new images? smile If so, then Scott's concern seems well founded. But not allowing submitters and reviewers to follow people seems like a harsh solution. What if instead, when you post an image, you have to explicitly click a button that opts the image "in" for eligibility. Once an image becomes eligible, all Submitters would get a notification for that image, just like the one you get when someone you're following posts a new image. I don't know how well this would scale, though, maybe you'd have to round robin to submitters instead of all of them being notified for every image. Anyway, I like the direction this is heading and wanted to give some feedback. smile
Ethan
15 Oct, 2016 00:38
Hearing lots of good things about this, both from the original post and replies. I wouldn't mind helping out, but I'll have to see if I actually can before typing the magic words!

Edit: I volunteer as a reviewer
Edited 15 Oct, 2016 01:19
siovene
15 Oct, 2016 07:52
John Renaud
Imho, there's a lot to like about this approach but Scott brings up a good point: how do Submitters become aware of images that should be evaluated for submission? Is the intent that they tirelessly and relentlessly browse and scan for new images?
AstroBin is no Facebook or Instagram with millions of images uploaded every hour :-) We're only talking about 100-200 new images daily. They can be browsed via the recent uploads page or I can make a dedicated page if it turns out to be necessary.
siovene
15 Oct, 2016 09:14
John Renaud
What if instead, when you post an image, you have to explicitly click a button that opts the image "in" for eligibility.
I suspect that would yield to a very high number of submissions  smile
siovene
15 Oct, 2016 09:19
Scott
The judges should not be allowed to follow anyone either.  In reference to the infamous thread from a year ago, has anyone actually clicked the "opt out" button?
Reviewers and Judges get their images to review and elect (respectively) from dedicated page, so whom they follow doesn't have any effect on that.
Submitters should check the "All recent uploads" page, or I can make a dedicated page for them if that proves to be not good enough.

All roles would be periodically shuffled around, and there would be incoming and outcoming people in different roles, to ensure diversity.

Me, or a board, will periodically inspect the activities of the various boards, to determine if some individuals are not performing sufficiently well.

Oh you're right about the "opt out" button. Only 29 users have checked it! smile
dvalid
15 Oct, 2016 13:47
Salvatore, I really appreciate your efforts to make the process fair and transparent. Though, the main problem was and will remain the final stage, IMO. Honestly, I do not believe that this complicated mechanism would have a great influence on the final results without solving the main problem. We all are human beings, with our subjective aesthetic approach and a personal ties.  So are the judges. You personally never failed to choose the best from the best, sadly that's not always the case..

 I strongly believe that the right to judge others is a huge responsibility at first place, rather than privilege. Increasing this responsibility is one of the most effective ways to minimise personal factors. Bringing out from the shadow of a person who is in charge for  IOTD was definitely a step up.  Why not make the next move in that direction? Allow us, the users, to be "judges of the judges". Say, let us anonymously(!) like or dislike the choice that judges have made. This will make obvious whether or not our community agrees with the decision.

Well, during this vote, some of us would be driven by emotions, but a picture in a whole would not suffer from that, imo. Benefits?
1.  I do not think that the judges would like to see that community does not approve their choice.
2.  If bias will be the trend for some of the judges, you'll be aware of that, with possible consequences..

P.S. I do not think that IOTD should always be the best picture, with dozens of hours of integration and perfect processing. Sometimes good enough effort from a beginner or a person with intermediate skills deserves to be encouraged. This will make our community only stronger.

CS!
Edited 15 Oct, 2016 15:18
planethunter
15 Oct, 2016 14:25
I think the new improvements sound great.  Might also be a good idea to make a restriction where the same person cannot keep winning the IOTD in a given time frame.
qbool
15 Oct, 2016 14:59
Salvatore Iovene
John Renaud
Imho, there's a lot to like about this approach but Scott brings up a good point: how do Submitters become aware of images that should be evaluated for submission? Is the intent that they tirelessly and relentlessly browse and scan for new images?
AstroBin is no Facebook or Instagram with millions of images uploaded every hour :-) We're only talking about 100-200 new images daily. They can be browsed via the recent uploads page or I can make a dedicated page if it turns out to be necessary.
I can only speak for myself and maybe it's just me but, if I were a Submitter, I would want to know that I'm fulfilling my duties as Submitter… if I have to rely on my own memory when looking through hundreds of images to determine which images I've already looked at and which I haven't, that seems like a recipe for images being missed. I routinely see images from people that I follow that seem like fairly high quality images but, yet, have a lower than expected view/like count b/c they don't have a lot of followers - this is a symptom of the problem that concerns me. This new workflow has the potential to get purposeful eyeballs (the Submitters) on these otherwise invisible but high-quality  images but I think, in order to make that happen, you have to give Submitters the tools they need to fulfill their duty, rather than relying on sporadic ad-hoc browsing. Having a notification mechanism specifically for the Submitters would be one way to accomplish that.  Just my $.02. smile
qbool
15 Oct, 2016 15:16
David Dvali
Salvatore, I really appreciate your efforts to make the process fair and transparent. Though, the main problem was and will remain the final stage, IMO. Honestly, I do not believe that this complicated mechanism would have a great influence on the final results without solving the main problem. We all are human beings, with our subjective aesthetic approach and a personal ties.  So are the judges. You personally never failed to choose the best from the best, sadly that's not always the case.. I strongly believe that the right to judge others is huge responsibility at first place, not a privilege. Increasing this responsibility is one of the most effective ways to minimise personal factors. Bringing out from the shadow of a person who is in charge for  IOTD was definitely a step up.  Why not make the next move in that direction? Allow us, the users, to be "judges of the judges". Say, let us anonymously(!) like or dislike the choice that judges have made. This will make obvious whether or not our community agrees with the decision.

Well, during this vote, some of us would be driven by emotions, but a picture in a whole would not suffer from that, imo. Benefits?
1.  I do not think that the judges would like to see that community does not approve their choice.
2.  If bias will be the trend for some of the judges, you'll be aware of that, with possible consequences..

P.S. I do not think that IOTD should always be the best picture, with dozens of hours of integration and perfect processing. Sometimes good enough effort from a beginner or a person with intermediate skills deserves to be encouraged. This will make our community only stronger.

CS!
Wouldn't this approach just defeat the purpose of having judges though? If negative pressure is put on a judge for a particular selection b/c of popular opinion, this would likely influence that judge's next selection in favor of a more popular image. If you follow down that path far enough, we're essentially back to like-driven selections, yes?

I hear what you're saying about bias… I think the proposed checks and balances would take care of that though - the board convenes periodically and evaluates performance/fitness of all positions and replaces as needed.
dvalid
15 Oct, 2016 15:45
John Renaud
Wouldn't this approach just defeat the purpose of having judges though? If negative pressure is put on a judge for a particular selection b/c of popular opinion, this would likely influence that judge's next selection in favor of a more popular image. If you follow down that path far enough, we're essentially back to like-driven selections, yes?
John, I think we all agree that the popularity of particular image i.e., amount of likes are in good correlation with the number of followers. That is not the case for IOTD, which is visible to everyone. Small amount of likes does not necessarily mean that people do not like the image, reason could be few followers and vise versa. If such a picture became IOTD, then let people (us) decide whether we like the choice or not. I do not see how it would lead to like-driven selection.
Edited 15 Oct, 2016 15:48
qbool
15 Oct, 2016 16:04
Perhaps I don't understand your proposal then. It's already the case that the community weighs in a judge's selection in the form of "likes after selection". When the community strongly agrees with a selection, the image gets more post-selection likes than when the community doesn't agree as strongly with a selection. I guess the question is: should that number be taken into account when evaluating the "performance" of a given judge? I don't necessarily think it should. That number might come into play if there's any question of bias for a particular judge… but, in general, my opinion is that different judges are using subtly different criteria when evaluating images and they should be - the more diverse the judging panel, the better if you ask me. Because of this diversity, it follows that there will some popular selections and some less popular selections. What's more important is that every judge on the panel be as impartial as possible (hey, we're all human) and not show any bias. I think that's all you can really ask for and expect from a judge. Again, just my opinion. smile
dvalid
15 Oct, 2016 16:35
I did not say that judge necessarily should face the consequences. But statistics will show how often particular judge deviates from general opinion.

BTW, that's a good idea to reset counter of "likes" for an image, selected as IOTD. The number of likes will show then how strong community supports the choice. In that case the "dislike" button is not needed at all. Of course, people who "liked" image before it became IOTD should have the right to like it again smile
Edited 15 Oct, 2016 17:05
siovene
15 Oct, 2016 20:07
planethunter
I think the new improvements sound great.  Might also be a good idea to make a restriction where the same person cannot keep winning the IOTD in a given time frame.
I can't say I agree with that. Good images are good images, regardless of who authors them.
siovene
15 Oct, 2016 20:08
David Dvali
that's a good idea to reset counter of "likes" for an image, selected as IOTD.
No need to reset them. Likes come with a timestamp, so I can count the likes that came after the election to IOTD, from the database, if we decide to give that suggestion a try!
Hondo
15 Oct, 2016 20:13
Will images from remote observatories count, ie: DSW, etc or any image that was not acquired by the user and their own equipment?
Goofi
15 Oct, 2016 20:29
I'm going to speak up - for myself only - and share thoughts on judging IOTD (as one of the current judges).

It's a huge responsibility, and one I take seriously.  It's also partly subjective and partly objective. Some images are good and some are not so good. You can review the group we have to sort through each day by clicking on the gold IOTD banner.  But, if I narrow the grouping down to a half-dozen good images, that still leaves a lot of room for subjective picking.

For myself, I review the images on their merits, and then I look at two other factors. Have I see this image style a lot lately? If so, I'll be inclined to choose something else. Also, have I seen this image class a lot recently?  As a result of these two subjective checks, I'm comfortable picking a solar image, or even a nightscape, as I don't see them in IOTD as much.

If I were to suggest one change to IOTD, it would be something a little different than the direction Salvatore is taking this. I'd make it so the judge who picks can do a short explanation of why that particular image was picked.  Every day there are a number of outstanding images to choose from, any one of which is deserving. I'd love to be able to share why the particular image was picked.  To me, that adds the transparency of judging.

I like things simple. I think adding in a pool of pickers and selectors is going to complicate things. But, I do see where it builds community and so it probably is a good idea. I just think if there's some way to get feedback from the judges why an image was picked it will help the community as a whole, too.
dvalid
15 Oct, 2016 20:31
Salvatore Iovene
No need to reset them. Likes come with a timestamp, so I can count the likes that came after the election to IOTD, from the database, if we decide to give that suggestion a try!

But without resetting that will put images with more "likes" gathered before they became IOTD, into unequal position - they will gain less likes after..
rob77
15 Oct, 2016 21:50
David Dvali
Salvatore, I really appreciate your efforts to make the process fair and transparent. Though, the main problem was and will remain the final stage, IMO. Honestly, I do not believe that this complicated mechanism would have a great influence on the final results without solving the main problem. We all are human beings, with our subjective aesthetic approach and a personal ties.  So are the judges. You personally never failed to choose the best from the best, sadly that's not always the case..

Exactly. I completely agree with you. Making things more complicated doesn't necessarily will help to solve the main problems
rob77
15 Oct, 2016 21:54
Scott
Will images from remote observatories count, ie: DSW, etc or any image that was not acquired by the user and their own equipment?

This is another good point.
Maybe few (just not to increase again the complexity) categories can be implemented:

- deepsky
- remote
- solar system

Cheers
rob77
15 Oct, 2016 21:59
Goofi
I like things simple. I think adding in a pool of pickers and selectors is going to complicate things.

Correct Goofi.
Stating that I thank a lot Salvatore for spending time on this topic, I go in the same direction as you. I don't like complicated stuff and this proposal, to me, sounds excessively tricky and doesn't guarrantee the solution of the problems.
I am honest to everyone,  I've been rarely IOTD lately so I don't have any reason to defend it, but I think that the current method (with some adjustments, like creating an offical judge pool that can even write a small review about the chosen image) is not so bad at all.
I have seen many beautiful images as IOTD recently.

Cheers
Edited 15 Oct, 2016 22:49
KuriousGeorge
16 Oct, 2016 00:55
How about keep it super simple. Hire 2-3 professional astrophotographers that don't post on this site. Then do image(s) of the week so you don't burn them out. Select from images posted over the last month or so. Could even go longer.
Edited 16 Oct, 2016 01:00
siovene
16 Oct, 2016 07:43
Guys, I know that the IOTD is a very polarizing topic and people have a million different opinions about it.

But we had a very long brainstorming topic last year where we discussed it to exhaustion.

Based on the ideas and points of interest expressed in that topic, what I presented here is the solution we have reached.

This is therefore not the time to propose radically different alternatives.

Instead, this is mostly a call for submitters and reviewers.

So who is up to actually help and make the IOTD better?

Please let me know here, or via private message if you prefer.

Thanks!
Salvatore
rob77
16 Oct, 2016 09:07
Salvatore,
it worths to say that you should make at least a binary easy and quick poll:

  1. go for the new approach
  2. remain with the old one
This would be very nice considering that you are changing the core feature of the site and, as others also stated, this change seems not to address the main problem which, for me, can be resumed here:

David Dvali
IMO. Honestly, I do not believe that this complicated mechanism would have a great influence on the final results without solving the main problem. We all are human beings, with our subjective aesthetic approach and a personal ties.

In totally honesty, I decided to go on with Astrobin, even paying when other services are free, because the "like" concept - even with some necessary modifications - jointly with a pool of judges that, as last instance, select the IOTD in my opinion was fairly working.

Now the scenario totally changes. Voting (i.e. putting "likes" to) images, for those who are not submitters, doesn't serve anymore.

Another point, ok, let's go with this scenario.
Why don't we add an automatic (based on some criterias that we can easily build here in the forum) feature that pre-select the images for the submitters?
Come on, submitters browsing alone astrobin limits to much their range of action considering that, at least from South America, the site is definetely slow (it takes seconds to open the main page sometimes).
Cheers
 
This topic was closed by a moderator.