We're always looking for ways to improve the experience on RepostExchange and recently we’ve been looking at Feedback Campaigns. 

One thing we've noticed is that the current system for rating feedback is inconsistent and often unfair.

To improve this we're introducing an AI-powered rating system, with moderator oversight, to evaluate feedback going forward.

We are also simplifying the Rep Score and Level system - we are removing the concept of Rep Scores, and your Level will simply be the average rating of your (up to) 10 most recent reviews.

How is the rating system changing?

Previously, we asked the person receiving feedback to complete a short questionnaire asking whether each piece of feedback meets certain criteria. The responses to the questionnaire produced a rating out of 6.

Going forward, this evaluation will be performed almost instantly using an AI system we have developed and tested over many months. 

Why is the rating system changing?

• The rating questionnaire experience was cumbersome and time-consuming
• User-generated ratings were not always objective or consistent
• Questionnaires sometimes took days or longer to be completed by users.

What are the advantages of the new AI-assisted system?

• Instant - feedback providers receive an instant rating for their response, which updates their level (and earnings) in real-time.
• Consistent - feedback is evaluated against a consistent set of criteria, and ratings are not biased by subjective views or emotions.
• Improving - with moderator oversight and user feedback we can continually improve the evaluation process to make it as fair and effective as possible

How does the AI-powered system work?

The AI rating system objectively evaluates the quality of written feedback. It does not make any subjective judgements about whether the feedback is right or wrong (because this will often involve opinion or taste).

As with the previous system, feedback will be scored mainly based on the following criteria: Constructive Criticism, Specific Observations, Knowledgeable Feedback, Suggestions and Solutions, Useful Resources and Requested Areas Covered.

The AI tool does not mark down feedback for spelling or grammar mistakes, or for different styles of writing, but it does look at levels of detail and how effectively points are expressed.

Will the feedback receiver be able to influence the rating at all?

While the evaluation and rating will be carried out by our AI tool, feedback receivers will be able to let us know if they think the rating is too high or too low. Human moderators will be monitoring this input closely in order to improve the system, and ratings will be manually adjusted if necessary. 

Additionally, users can report feedback using the flag icon if it appears nonsensical, irrelevant or abusive. 

How is the Level / Rep Score system changing?

Previously, each rating a user received modified their Rep Score, the Rep Score decided the Level and the Level decided the amount of credits a user can earn per feedback.

To simplify the system, we are removing the Rep Score element. Instead, a user will simply have a Level which is calculated from their average rating from their previous 10 reviews.

How will you transition to the new system?

Your current Level will become your average rating. For instance, if you are currently Level 5, the system will assume your previous 10 reviews had a rating of 5, giving you an average rating of 5.0. 

Going forward, every new rating will affect your average as would be expected.

What does this mean for new feedback providers?

New feedback providers will start on Level 3. The average rating will be calculated from after your first response onwards.

Why is there a daily limit on feedback I can provide?

While we test the new system with members, this 5 response per day limit will remain in place to help us effectively monitor responses and user behaviour.

Why was my campaign stopped?

As we transitioned over to the new system on 21st February, all running campaigns were stopped. Any unused credits were refunded to your account so feel free to create a new campaign. We apologise for any inconvenience.

How can I provide feedback, suggestions or bug reports about the new system?

Please contact support@repostexchange.com, we’d love to hear your thoughts!