Over the last 24 hours the Envato Market API has been incorrectly serving HTTP forbidden errors for all private access endpoints (such as purchase verifications). While we have already restored the service to full operation we understand many authors depend on this functionality for external services and want to apologise for the disruption. We understand any downtime is too much and we are working on ways to make the API more resilient and reliable to ensure this doesn’t happen again.
On the 17th of March, we deployed a change to the API that tightened the security checks performed when requesting private API endpoints. On the surface these additional checks coincided with our understanding of how the API request flow was working and would not pose any problems for legitimate requests. Our understanding was incorrect, and we instead caused all requests flowing through this code path to be rejected despite having correct permissions.
Unfortunately this regression wasn’t identified until we started receiving support tickets and emails stating that authors were having issues verifying purchases.
Why did it take so long to fix?
While we do have monitoring systems that keep watch over the API, they are not fine-grained enough to report on individual endpoints - instead, we detect service outages. This monitoring did not alert us, because the API itself was not unavailable.
What are we doing about it?
We have already implemented a short term solution for monitoring the affected private endpoints to ensure that we don’t introduce any similar problems. In the coming weeks we are looking to extend our monitoring suite to include the OAuth and Token based request flows for each endpoint which will be scheduled to continuously run against our production infrastructure using multiple geographical regions and user profiles. This will help us identify any issues within minutes after a deploy instead of needing manual verification or user to report it. We also intend to expand our integration tests so we catch this sort of issue before users ever see it.
If you have any questions or comments, please don’t hesitate to let me know and I’m happy to answer them.