February 2017

Delays in toutapp.com
Data display is now populating accurately. Happy Touting!
Feb 1, 12:50-22:56 PST

January 2017

Degraded Performance - Search
Everything is back to normal now. Happy Touting!
Jan 31, 15:31-18:49 PST
Analytics Latency
Analytics is fully up to date. This issue has been resolved.
Jan 19, 14:29-19:59 PST
GMail Service - Down
GMail Service has been running normally for the past several hours. Thank you for your patience and Happy Touting!
Jan 18, 13:44-20:13 PST
Degraded Performance
Earlier this morning beginning at 8:45 we experienced a brief period of degraded performance In our core database, resulting in increased page load times and background job processing times. We immediately deployed a fix, and everything is back to normal now. Happy Touting!
Jan 18, 09:25 PST
Degraded Performance
All background jobs that were delayed have completed processing and performance is back to normal levels.
Jan 17, 14:37 - Jan 18, 05:55 PST
System Outage
Our engineering team deployed a fix and we are back up and running. Happy Touting!
Jan 12, 17:51-17:55 PST

December 2016

Analytics Service - Data Latency
Our data reload script has finally completed and our Analytics pages are successfully loading real-time data. We will publish a post-mortem tomorrow after our Engineering and Operations teams perform a complete review of the root cause and resolution of this incident, and monitoring around our Analytics service. Happy Touting!
Dec 19, 08:40 - Dec 20, 17:16 PST
Degraded Performance - Search
Performance has returned back to normal levels. Happy Touting!
Dec 8, 16:17-17:03 PST

November 2016

Degraded Performance - Background Processing Jobs
Background job latencies have returned to normal levels. Happy Touting!
Nov 17, 17:45-18:10 PST
Degraded Performance
Response times have returned to normal levels site-wide.
Nov 17, 17:29-17:34 PST
Degraded Performance - Search
Performance across our Search Service has returned to normal levels. Happy Touting!
Nov 17, 09:05-11:09 PST
Salesforce Outage - Remediation
Yesterday evening starting at 4:23PM PST, Salesforce experienced a service disruption across all instances in their Dallas data center that lasted until 7:55PM PST. These instances include NA7, NA8, NA32, NA37, NA42, NA43, NA48, CS13, CS30, CS50, CS51, CS62, and CS63. More detail on this incident can be found here - https://status.salesforce.com/status/NA8/incidents/304. While ToutApp did not experience any disruption in service, all messages that our Salesforce Integration Service attempted to send to Salesforce for customers using one of the above instances failed during this time. All messages that failed as a result of the Salesforce downtime will be resubmitted later today to ensure that Salesforce and ToutApp remain in sync.
Nov 17, 06:20 PST
System Outage
Background jobs are caught up and back to normal. Thanks for your patience and Happy Touting!
Nov 10, 09:20-10:12 PST

October 2016

Email Delivery Service - Delayed Performance
The incident with our Email Service Provider has been resolved. Any messages that were delayed have been processed.
Oct 31, 08:58-11:04 PST
Knowledge Base and Ticket Submission are Down
This incident has been resolved.
Oct 21, 10:55-15:46 PST
Help Center is Down
This incident has been resolved.
Oct 13, 11:36-14:04 PST
Analytics Down
The Analytics Tab is back up and running. Thanks for your patience.
Oct 13, 09:52-09:59 PST
Performance Degradation
The Performance Degradation issue from this morning has been fully resolved. Thank you for your patience and Happy Touting.
Oct 12, 06:15-11:38 PST
Monitoring Blip
The downtime that appeared earlier this morning was just a blip in our monitoring infrastructure service that is now resolved. We have been fully operational all morning.
Oct 10, 10:37 PST

September 2016

Email Delivery Service - Delayed Performance
We were notified by our Email Delivery Service provider that the performance issue resulting in intermittent delays in email delivery that may have impacted some of our customers has been resolved at 11:07 Pacific Time.
Sep 27, 08:41-11:47 PST
Email Delivery Service - Delayed Performance
We received notification from our email provider that the issue from earlier has been resolved.
Sep 26, 08:20-09:32 PST
Performance Degradation
Script is now finished and the site is operating normally! We will be continuing to perform optimizations over the next 24 hours.
Sep 21, 11:25-18:13 PST
Email Delivery Service - Delayed Performance
This incident has been resolved.
Sep 1, 10:40-16:51 PST

August 2016

ToutApp Service Down - Monitoring
All clear!
Aug 3, 08:56-09:58 PST

July 2016

Analytics for today showing as 0
analytics dashboards are now fully caught up
Jul 6, 13:08-15:05 PST

June 2016

Service Interruption
And we are back in business.
Jun 16, 17:28-17:30 PST
Analytics - Degraded Performance
At 1:15PM our operations team added additional servers to our Analytics infrastructure and resolved the performance issue with the Analytics service. We have seen performance levels return back to normal since then and will continue to monitor the service.
Jun 14, 13:08-14:49 PST
Degraded Performance
All systems are now operating normally. Thank you for your patience!
Jun 13, 09:21-14:47 PST
Performance Degradation
The performance degradation we experienced from earlier this morning has been resolved. Thank you for your patience.
Jun 13, 05:46-06:46 PST
[Scheduled] Infrastructure Upgrades - 6/11/2016
The infrastructure migration has completed and our site is now operational. Happy Touting!
Jun 11, 14:41-18:05 PST
Service Interruption
The issue from earlier this evening has been resolved. All systems are operational
Jun 9, 17:43-17:49 PST

May 2016

[Scheduled] Routine Infrastructure Upgrade
The scheduled maintenance is complete! Thank you for your patience and Happy Touting!
May 1, 10:00-10:50 PST

April 2016

Delay in Analytics data
We have fully revalidated the Analytics platform to ensure accuracy. All systems operational!
Apr 25, 11:42 - Apr 26, 05:11 PST
ToutApp Email Servers Delayed
Delivery queues have now emptied!
Apr 25, 12:06-14:42 PST
Brief Downtime Incident - Resolved
We experienced 7 total minutes of intermittent downtime earlier today as a result of a series of slow database search queries. The root cause was addressed and the incident was fully resolved by 10:33AM PST.
Apr 13, 15:39 PST

March 2016

Usage Degradation
At 4:31pm PDT we experienced a rapid increase in error reports which caused various actions on the site to fail. This was caused by an unexpectedly cached DNS address in a portion of our underlying search infrastructure. The issue was resolved at 4:45pm and by 4:50pm all systems are fully operational.
Mar 29, 17:06 PST