TL;DR:
- The review queues continue to be the focus of a great deal of work:
- We have made inroads into improving the capacity of our team and infrastructure to be able to deal with higher volumes of items
- The rate at which new items are being submitted also continues to rise meaning that wait times for new submissions continues to increase.
- We have a roadmap of several initiatives that we detail below to bring the queues down further. Specifics relating to these initiatives and challenges can be found below.
Hi everyone,
We’re now approaching the end of June, so we wanted to jump into this thread and give you all an update as to where we’re at, some of the changes we’ve made, the issues we’re facing and the roadmap moving forward.
Firstly, let’s talk about the current state of our queues. For the purposes of giving you context around what we’re doing, we wanted to paint a clearer picture of how the queue works:
Item updates, resubmissions and submissions from Power Elite Authors are moved to the front of the queue. I’ve listed the reasons why we assign high priority to these types of submissions below.
Updates: This comes from a need to maintain the quality of current items on the market. Updates address a range of important things, from ensuring items are compatible with Wordpress Updates, to fixing a bug, or remedying a security vulnerability. We prioritise these because our first responsibility needs to be ensuring that the items that are on our shelves are safely maintained for our authors and our customers.
Resubmissions: If we’ve reviewed and soft rejected an item with feedback, then we need to be able to get to that item again as quickly as possible. Note: We recently increased the number of soft rejections.
Power Elites: This has historically been a reward we provided to our authors once they reached the 2M+ mark to recognise their high value contributions. It has since expanded to include the 1M+ authors as well. These submissions account for a very, very small number of items overall.
So how does this look in practice? Below are some graphs that illustrate the WordPress Category on ThemeForest.*
*Note, “Reviewing Actions” refers to the number of reviews undertaken. As “New Items” is the only category in which demand exceeds supply, we have broken this into “new items” and “new items submitted”. The former is the number of new items reviewed, the latter is the number of new item submissions.
Firstly, let’s begin with the obvious; the wait time for new WordPress items is averaging sixty days. We’re a long way from where we want to be in this area, and there is a fair amount of work in our path before we arrive.
We need to openly face an uncomfortable truth: Whilst we are doing everything we can to try and reduce the wait times for new items, the actions we are taking will not have immediate impact. Things will improve, but it will take time, and things will likely get worse before they improve. It could be another couple of months before we see a reduction in wait times for new items.
We know many authors are waiting for new submissions to be approved, and in some cases, have a great deal riding on those submissions. For that, we can only offer our heartfelt and sincere apologies.
We are now focused on doing everything we can towards repairing the situation. Below, I’m going to talk in specifics about the work that we’ve done thus far, and what our roadmap is moving forward.
1. We’ve more than doubled the size of the ThemeForest review team.
We now have a fresh group of reviewers in the process of being onboarded to our team who are learning the ropes.
2. We’ve increased the size of the AudioJungle review team by 36%, to reach almost 50% over the course of the next month.
They too are in their early days of learning how our system works, but as they get up to speed, our capacity to review greater numbers of items will increase.
3. Our engineers have made some headway into resolving the enormous infrastructure challenges that we mentioned in earlier posts. Let’s get specific:
- Auto-approval for qualified authors (due to be re-instated in the next few days)
- Getting content to our reviewers faster to speed up review times. We’re close to completing a test of this, which is intended to speed up the average time to review an item. We expect to have some results in July
- Trying ways to re-prioritize the queue (This didn’t work too well, see below for the challenges with the code)
- Created tools to speed up review of updates to WP themes by detecting changes between versions. These are already in use by our review teams.
- We’ve included all Power Elite Authors (1M+) in priority reviews.
We are also facing some serious challenges.
- The code that our team is trying to work with, is, putting it mildly: incredibly complex. In some places, it’s nearly 10 years old. To give just one example: When a new preview is uploaded for a VideoHive item, there are four totally different sub-tasks that the system creates and has to keep track of, and three of them are tracked in totally different ways. Working with this stuff is incredibly complex and time consuming for our engineers. Many of you have already noticed that within the scope of our work we’ve encountered problems that have triggered unintentional side-effects.
- Whilst our capacity to manage the number of submissions is increasing, so too is the number of submissions that we’re dealing with.
- In some categories such as WordPress, items are becoming increasingly more complex and multi-purpose. This increases the time it takes us to review both new submissions and updates to existing items.
Which brings me to the roadmap of how we’re going to tackle this moving forward and the specific initiatives we have planned:
1. We will continue the process of hiring new reviewers to better bring supply to the level of demand.
We need to ensure our reviewers are supported and given the time they need to get up to speed. As they learn more, and become more efficient, the rate at which we can review items will increase. Quality control here is incredibly important, both in the recruitment of the reviewers (we’re quite picky) and the training we provide them.
2. We are continuing to focus on improving the tools and infrastructure that support our reviewers to do their work.
Much of this is around bringing in more automated checks on items to ensure our reviewers can get through more items, faster. These will steadily chip away and bring incremental improvement over time.
3. Encouraging higher quality submissions of items
One of our biggest challenges is the amount of time we spend reviewing items that are never approved for sale. To illustrate, approximately 75% of new items submitted to ThemeForest never reach a state of being approved by our Quality Team. We’re proud to be a marketplace that allows anyone to have a chance to succeed, but want to look towards ways to raise the quality level of items before they reach the point of being submitted. Over coming months we’ll be bringing in changes to the author experience to encourage submission of higher quality items, priming authors for success.
5. Better reporting on queue wait times
Although having better reports does not in itself improve the wait times, we want to improve the way we communicate wait times to our authors. Currently, we only display queue averages on our http://quality.market.envato.com portal. Whilst the average wait time is useful, it risks generalising as depending on the nature of the item and the author, there is variance. We know that better context gives our authors the power to make better decisions.
It is the unfortunate reality of our work that we cannot clearly say when the review queues will begin to drop. There are just too many unknowns. All we can do is focus our energy and attention on improving our team’s capacity to manage the queues.
We read each forum post and know well the pain that review queues are causing many authors, but we also wanted to take a moment to give heartfelt thanks to the members of our community who taken the time to express their support for us. Whether through applying to become a reviewer, a supportive email or simply a forum post, we are grateful to have you with us.
We will continue to keep you updated.