The subject of Review Consistency regularly comes up on the boards again, so we felt the time was right for the matter to be addressed, for what it’s worth.
Based on the astute observations of some of our long time members (ahem… privet, Kurly), we can either deny the existence of inconsistency outright, stonewalling, or treat it as a reality that needs to be managed in an ongoing way.
But we’ll say it again, we care about what you think, so let’s begin by admitting the truth.
At the end of the day, it would be silly to steadfastly proclaim that inconsistency never exists here.
That would be simply untrue.
Absolute consistency is an abstract concept that applies only in theory, not in practice, so long as people are making decisions in the complex world of appraising the objective commercial potential of art - a subjective medium by definition.
So there. We can say this very, very clearly. Inconsistency unquestionably exists in some areas of the review process, and always will. In fact, believe it or not it’s something that needs to be managed on a regular, near daily basis. It’s not something we hide from, internally, as all reviewers know. But do we owe it to you to tell you a little bit more about it? That’s a debatable point, for a business, but I’d like to think it might help any annoyed people understand what happens, and why, even if it will not change the outcome much soon.
At this point, the most important question to answer, to deal with managing inconsistency is - How does it manifest itself, and why?
Let’s dig a little deeper to get to the core of the issue, to see what we can learn.
I. Team Inconsistency
First off, we currently have a team with 11 awesome individuals listening to submissions. The people reviewing your tracks are ALL well-meaning and well intentioned, driven by a sense of responsibility toward the quality of the library and an honourable respect toward the community of authors. If anyone has any doubts about this, or feels there is any conflict of interest or personal vendetta, the fact of the matter is you don’t know each person on our team well enough, simply put.
Please keep in mind that we do have a system that aims to minimize inconsistency on a daily basis. However we will never be able to eradicate it outright, so long as people, especially groups, are deciding things
What we can tell you for starters is that inconsistency is practically non-existent in the extremes. Even with 11 people. That means the very best and the very worst submissions. In the absence of strong misrepresentation indicators, stellar submissions are practically always accepted, and categorically inadequate ones are rejected. We can agree on this, yes?
Now everyone can hear how great some of the best tracks we receive sound, but unfortunately the very worst of the lot never see the light of day. And we get a lot of of both. We’d really love to give you examples of the “best of the worst” we hear, for your elucidation, if anything, but for privacy reasons we simply can’t. But we have a feeling most of you wouldn’t believe your ears.
If you can picture as an example, a track that sounds like a locomotive having intimate relations with a chainsaw while a music box sample plays in the background in a completely different key, with birds singing here and there, and random atonal electronic vocal accents… Well, you get the picture. Very creative perhaps, but AudioJungle is not the right library for that! 
Forgive the digression… Back to the point.
The biggest incidence of inconsistency happens with borderline tracks that are submitted to us.
Wait… What? What is a borderline track??
We thought you’d ask. To answer that, first we need to understand what “borderline” means, as far as we’re concerned.
Here’s the thing - “Borderline" is not a line at all. It is a Spectrum
In this sense, a borderline track is one that falls somewhere closer to the middle. A borderline track does not stand out as being exceedingly flawlessly executed and/or with commercial potential, for any number of composition, arrangement, or production reasons, but its flaws do not immediately make it sound completely or more extremely lacking, for general commercial purposes.
Essentially, that’s where we’ll experience the most inconsistency when we review. In that middle third of the gray zone. And the closer to the middle gray a track falls, the more likely it can be inconsistently reviewed.
But How does that happen? And why?
Great questions.
If a track is very borderline, the bottom line is it can go both ways, theoretically, and that also may depend on a number of factors…
As a human reviewer, previously heard content may variably influence a borderline decision regardless of the training and experience. It is the human nature of habituation.
That’s what can make a “3” or “4” grayscale feel like a “5” or “6” grayscale, or the other way around
Imagine you heard several great tracks in a row right before hearing a more average one… The borderline WILL sound worse to your ears. The middle spectrum can shift away from you.
Yet If you heard several entirely unacceptable submissions immediately first, the borderline may get approved. The shift may also happen in the other direction
Throw in a bunch of different people in the mix at different points in their shift, and you get an idea of how things can get hazy. And yet there is strong training to remain focused and disciplined. In many cases the second reviewer may choose to defer to the first one, but this is not always possible obviously.
Now if a reviewer accepts a track that could arguably be rejected, (barring system errors) it’s significantly more often than not a borderline track, that may or may not add value to the library.
Vice versa, if a reviewer Hard Rejects a track that could arguably be accepted, well, it boils down to the same thing.
Ok, so how come my track was accepted when I resubmitted With NO changes??
Before saying anything else on this, we will just clarify - If you resubmit a Hard Rejected track without making any changes, (1) You are breaking the rules and (2) You are taking a risk that grows each time you repeat this.
Let’s assume you throw caution to the wind. If that rejected track is resubmitted with no changes, several things can happen.
1 - Most likely, the track will simply be rejected again, if it was on the lesser borderline side, objectively and subjectively.
2 - If it is matched by the system as a resubmission with no changes, it may or may not be reviewed again by another reviewer. The account may get flagged in our database either way.
3 - Less likely, statistically, if it is determined that the item was acceptable on the second pass, it may get accepted. In this case the first reviewer may get a notice and a consistency advisement may be dispatched to the team for ongoing training. The account may still be flagged for resubmitting hard rejected content
4 - Also less likely, if the rejection is accepted the second time, but the second reviewer accepted incorrectly, the second reviewer may get the notice and a consistency advisement may also be dispatched to team for training purposes.
5 - Believe it or not, we also have instances where a borderline track that should be rejected gets accepted on the First pass, and it only gets noticed when an update is submitted, or otherwise. While this exemplifies and publicizes the notion of inconsistency when held under scrutiny, and we know people are watching, we won’t usually overturn those if there wasn’t a legitimate processing error, even if it creates the impression of more inconsistency! But the reviewer and team will still receive private advisory and notes for training.
It really needs to be understood how much awareness there is on the matter, and there are always ongoing efforts to counter significant imbalance in consistency.
To say that many rejected tracks are accepted when they are resubmitted unchanged is a relative statement. While the number is obviously not zero, it is a very minor proportion when looking at percentages. That much we can tell you without a doubt. So while the statement is not wholly untrue, it is really a much more statistically minor fraction, albeit one that can attract visibility we readily admit.
Ultimately, Resubmitted Hard rejections will be usually hard rejected again when observed, accounts flagged, and if an emergent pattern of the practice is verified, whereby it is becomes evident that there is a habit of resubmitting hard rejected content without any changes or reasonable communications to the team, the account’s upload submission rights will be revoked, pending further communication via support channels. We don’t really have much of a choice when comms are repeatedly ignored.
Bottom line, do the referees see each and every foul that happens on the football field? You know the answer answer to this.
But when a pattern of infringement is verified, well, we have to have yellow and red cards too. That’s just the way it is. So please govern yourselves accordingly.
The regrettable part, in all honesty, is that resubmitting rejected tracks with no changes, accepted or not, does nothing to improve one’s skill at audio production. It just demonstrates a conceivable short-sightedness, where being accepted is more valued than producing content that could potentially sell more. Way more often than not, it devalues both the library and the portfolio, with a borderline content that is less likely to get sales AND make the rest of the portfolio less likely to be browsed further too. Please think about that, people.
And that’s why we have to draw the line sometimes.
II. Inconsistency over time
A separate issue, still worth mentioning while we’re on the topic, is the appearance of inconsistency of the approval standard as the library has evolved over the years.
We won’t delve as deeply, but the truth of the matter is “Acceptance standards have continued to evolve over the months and years, so something that may have been previously accepted in the past would not be accepted today.” Yep, that’s coming straight from the textbook.
This is why it is incorrect to make assumptions or comparisons with any items that were accepted 1-2 or more years ago, when considering submitting content today. Doing so could definitely engender a strong evidence of inconsistency. But it does not reasonably apply today. Apples with apples, oranges with oranges, please.
Is there a plan for this subset of the issue in the future? Yes, but this post is not the place to expound upon that. Something to discuss later this year perhaps
How can inconsistency be reduced?
Finally, winding to a conclusion, we can share a last few important points. We already do have consistency measures in place. They can never be perfect but the fact that they exist validates the awareness of the issue, internally.
In the queue, many of the items that we place on hold are ones that go through a second review without you realizing it. However, for efficiency reasons it’s simply not possible to assign 2 or more people to each submission received. We also cannot create multiple filtering systems now without Hugely impacting the length of queue. And then we’d be lambasted for that right?
But we do encourage our team to honestly seek another ear when doubts are felt. This minimizes inconsistency in many ways more than immediately realized.
We also scan the rejection forum threads with Community Team, and take appropriate action when warranted. Envato does believe in the value of “Fair Go”, as much as possible, after all.
But sometimes, to reduce inconstancy in the future, and as authors, what’s best called for is some really brutal self-honesty. When we ask ourselves: "Does my work stack up? Am I in any way closer to the borderline than I think I am?”
This is a question for all authors, and also for reviewers.
Anyone can always look for any ways that their work might be less adequate, or where any improvements are possible, but unfortunately as individuals we are all potentially prone to getting hindered, without even knowing it, by what’s known as the Dunning-Kruger Effect. It’s not a plugin. 
It’s an inescapable quandary of consciousness that can creep up in individuals of all levels. But at the end of the line, through persistent trials and tribulations, many believe it’s something anybody can surmount if there’s an open and honest will to do so, when looking inward upon oneself. And when each next level is reached, a veil is lifted and we can often see what we simply were not able to be aware of before. Because we are all sometimes blind to ourselves in the present.
If anyone can sincerely and introspectively admit to themselves that their submissions may be more so classified as borderline work, and they wish to minimize the experience of inconsistency, for themselves, they can either make efforts to improve their craft with time, patience and persistent dedication, growing as an artist, or face the fact that they will inevitably be subject to the inherent reality of inadvertent inconsistency in the review process of borderline submissions.
The review team is striving to do the same too, every day.
In any field, the other regrettable option is to remain annoyed or embittered and brazenly point fingers at the world for being what it is. But that doesn’t change it alone. Never will. It’s like cursing the ocean for making you wet each time you try to swim in it.
As the old saying goes, “if you’re not part of the solution, you’re part of the problem”.
While we cannot explain to all authors every single dynamic of the backend review system and expect them to suggest realistically deployable methodologies, part the solution can start with oneself. The only part we can control for ourselves.
By making every effort and using every avenue available to learn and get a bit better each day, the measure of inconsistency will be narrowed as two groups take steps toward one another, author and reviewer. But each group can only reach a midway point, so the actual measure between the two parties is dependent on the steps both parties take.
That leads to progress, and progresses beyond the realm of the borderline, and That, regardless of any undeniable inconsistency or imperfection of the process, will undoubtedly make the results more consistent because rejections will gradually occur less and less.
You can take it or you can leave it, but that’s how it happens here.
Have a great Sunday, forum friends, thanks for taking the time.
===============================================
Follow-up answers to questions, Why hard rejection feedback is not possible: Part 1 and Part 2
