~2:30PM: A new release is deployed. It had many different changes to various bugs throughout the system, including some updates to a new feature.
~2:55pm: Release succeeds.
3:00pm: Operations team begins to notice general slowness throughout site.
3:04pm: Issue is escalated, 502's begin appearing on webapp
~3:10pm: Initial investigation determines that a bad database update was made. Problematic entry is removed servers are restarted.
~3:15pm The 502 Errors continue.
3:38pm: It is determined that a new piece of code that looks up data for the aforementioned new feature is slowing down the site.
3:40pm: New Deploy is sent. Some site functionality is restored.
4:16pm: We determine that some of this bad code is still live running in production.
4:18pm: We stop all processes for this code on all servers.
4:24pm: Deploy is successful