On Friday August 31, for a window of just over three hours, the Envato API was incorrectly configured. This API is used to deliver notifications, earnings updates and other small applications. During the window, some users received incorrect API responses containing other users’ data.
It’s important to note that sensitive information such as login credentials or payment information was not leaked. Such sensitive information is not stored in this system nor does this system have access to it.
We know our community places trust in us to securely manage this data and on this occasion, we have fallen short on this expectation. We are treating this event and its resolution with the utmost importance and we apologize that the incident occurred.
The Event
At 06:41 GMT a change was deployed to our API caching infrastructure that altered how we determine cache keys and handle cache negotiation. This change was initially deployed earlier in the week to a tiny fraction of our traffic as a canary test and passed all of the testing we had in place to ensure correctness. Unfortunately the problem only appeared when there were significant numbers of users making requests. Our test suite did not exercise this case.
Within the next couple of hours, a support ticket and two forum threads emerged that described API users receiving incorrect responses. An Envato team member saw these and notified one of our on call engineers. At 09:56 GMT, the on call engineer quickly identified the issue and disabled the changes returning the API to normal operation. Due to the way we had rolled out the changes, the cached items had a very small time to live (TTL) which meant within, at most, 60 seconds all of the incorrect cached items would have self-expired.
Since the event we’ve conducted an analysis of the initial problem and impact. The vast majority of our API requests during the period were between confirmed automated sources, such as for theme verification. In these instances, the API operation simply failed rather than serving incorrect information.
Some requests did potentially expose information to users. The types of information transmitted included author earnings statements, lists of items purchased, numbers of item sales, details of items, usernames and email addresses. While purchase listings could have been returned incorrectly, the API endpoint for purchase verification was not impacted, ensuring that no user would have been able to activate a product they didn’t own.
Given that the exposure was for a short window of time, that no login or payment information was in this system, and that the majority of API requests were to automated sources, we do not believe there is cause for concern for our community. However we take these incidents very seriously, and are working hard to ensure we prevent similar incidents in the future.
Future work
We have put significant effort in understanding the cause of this incident and how we are going to mitigate such circumstances going forward. While we have identified the misbehaving code that resulted in an unexpected code path being taken, we know addressing this is only part of the long term solution.
Our engineering teams rely heavily on automated test suites to assert correctness and prevent regressions from making it into production. Despite having tests in place for this functionality, they failed to identify the problem and allowed the change to remain in production unnoticed. We’re updating our test suite to be more robust with the intention of exercising more failure code paths and handling it more gracefully in the event they happen.
Conclusion
We will continue to analyze the events surrounding this incident and use our investigation to improve the systems and processes that we use at Envato. Once again, we apologize that this incident occurred.
FAQs
Do I need to reset my password?
No. Passwords were not impacted by this event.
How do I find out what data of mine was compromised?
If you think you may have been impacted, please email help@envato.com.
