tag:status.foxpass.com,2005:/historyFoxpass Status - Incident History2024-03-29T00:53:07-07:00Foxpasstag:status.foxpass.com,2005:Incident/189266252023-10-25T11:00:00-07:002023-10-25T11:55:09-07:00EAP-TLS failing for some customer sites<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>Resolved</strong> - A small subset of customer sites cannot authenticate with EAP-TLS</p>tag:status.foxpass.com,2005:Incident/172432342023-05-20T22:17:08-07:002023-05-20T22:17:08-07:00Database Maintenance<p><small>May <var data-var='date'>20</var>, <var data-var='time'>22:17</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>21:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>22:51</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.foxpass.com,2005:Incident/172391522023-05-13T22:50:21-07:002023-05-13T22:50:21-07:00Database Maintenance<p><small>May <var data-var='date'>13</var>, <var data-var='time'>22:50</var> PDT</small><br><strong>Completed</strong> - The maintenance has been postponed.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>21:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>09:10</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled DB maintenance during this time</p>tag:status.foxpass.com,2005:Incident/171377232023-05-13T09:07:55-07:002023-05-13T09:07:55-07:00Database Maintenance<p><small>May <var data-var='date'>13</var>, <var data-var='time'>09:07</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>09:07</var> PDT</small><br><strong>Update</strong> - This was scheduled at the wrong time; should be 9pm PST not (9am)</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>17:16</var> PDT</small><br><strong>Scheduled</strong> - We will be upgrading our primary database during this maintenance window</p>tag:status.foxpass.com,2005:Incident/107548962022-08-03T14:00:33-07:002022-08-03T14:00:33-07:00Long-running LDAP connections return an error code for 'search' commands.<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>14:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>13:59</var> PDT</small><br><strong>Update</strong> - This has been resolved.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>12:58</var> PDT</small><br><strong>Investigating</strong> - We are actively working on this incident. In the meantime, forcing a re-connect (restarting sssd or nslcd) should create new connections which will not be affected.</p>tag:status.foxpass.com,2005:Incident/51530372020-09-22T18:07:02-07:002020-09-22T18:07:02-07:00LDAP binds are sluggish for some requests<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>18:07</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>12:01</var> PDT</small><br><strong>Monitoring</strong> - A small % of LDAP binds are slow to respond. We are working to improve the performance.</p>tag:status.foxpass.com,2005:Incident/50274672020-09-06T01:01:18-07:002020-09-06T01:01:18-07:00Database maintenance<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>01:01</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>22:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>21:51</var> PDT</small><br><strong>Scheduled</strong> - We will be upgrading our database. During the maintenance, the database will be read-only. Log events will be queued.</p>tag:status.foxpass.com,2005:Incident/49722582020-08-30T00:09:12-07:002020-08-30T00:09:12-07:00Database maintenance<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>00:09</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>22:13</var> PDT</small><br><strong>Scheduled</strong> - Database will be in read-only mode for about an hour; logs will be queued and written when the database returns to service.</p>tag:status.foxpass.com,2005:Incident/46904632020-07-23T08:45:37-07:002020-07-23T08:45:37-07:00Google Sync Errors<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>08:45</var> PDT</small><br><strong>Resolved</strong> - Google has rolled back the code that caused the error. We are not seeing any more issues.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>08:00</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>08:00</var> PDT</small><br><strong>Identified</strong> - We are getting errors from Google when trying to access their APIs currently. We have escalated to their support.</p>tag:status.foxpass.com,2005:Incident/37090102020-03-19T08:50:20-07:002020-03-19T08:50:20-07:00Service disruption<p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>08:50</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. Root cause was a failure of our primary database. Automated failover worked as designed and the system recovered about 2 minutes later.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>08:34</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.foxpass.com,2005:Incident/34189122020-01-12T02:21:36-08:002020-01-12T02:21:38-08:00Database Maintenance<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>02:21</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>02:21</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>02:21</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>02:15</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>02:11</var> PST</small><br><strong>Scheduled</strong> - LDAP and RADIUS will be unavailable for a few minutes while we do some required Database Mainenanace</p>tag:status.foxpass.com,2005:Incident/31331782019-10-22T19:08:16-07:002019-10-22T19:08:16-07:00Console outage<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>19:08</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>17:31</var> PDT</small><br><strong>Identified</strong> - This is the result of an AWS Route53 DNS issue. More details here: https://status.aws.amazon.com/</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>17:25</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue</p>tag:status.foxpass.com,2005:Incident/28867762019-08-19T13:31:56-07:002019-08-19T13:31:56-07:00G Suite Outage<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>13:31</var> PDT</small><br><strong>Resolved</strong> - G Suite is reporting that the issue has been fixed and we're seeing recovery as well.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>12:50</var> PDT</small><br><strong>Monitoring</strong> - We're seeing error rates improve and continue to monitor the situation.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>12:37</var> PDT</small><br><strong>Identified</strong> - Google G Suite is having issues signing users in and checking their passwords for delegated authentication. We are monitoring the situation.<br /><br />https://status.cloud.google.com/incident/developers-console/19008</p>tag:status.foxpass.com,2005:Incident/27279962019-07-13T10:47:21-07:002019-07-18T09:28:34-07:00Increased Error Rate<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>10:47</var> PDT</small><br><strong>Resolved</strong> - A misconfigured container caused in increase in LDAP error rates</p>tag:status.foxpass.com,2005:Incident/22924992019-03-27T17:44:18-07:002019-03-27T17:44:18-07:00Delegated authentication failures<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>17:44</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>17:32</var> PDT</small><br><strong>Investigating</strong> - We are seeing slow or failing authentication to third-party sources. We are investigating.</p>tag:status.foxpass.com,2005:Incident/19764632018-10-17T11:59:25-07:002018-10-17T11:59:25-07:00Service Outage<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>11:59</var> PDT</small><br><strong>Resolved</strong> - The service has recovered and is operating normally.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>11:44</var> PDT</small><br><strong>Monitoring</strong> - The primary database failed and a secondary automatically took its place within a few minutes. Service is recovering and we are monitoring performance.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>11:31</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.foxpass.com,2005:Incident/18447092018-08-03T15:30:55-07:002018-08-03T15:30:55-07:00Some EAP-TTLS RADIUS connections not succeeding<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>15:30</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved but requires action if you connect to our cloud-RADIUS servers for 802.1x. If you did not receive instructions via email, please contact us.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>12:49</var> PDT</small><br><strong>Identified</strong> - This appears to be an incompatibility with an update to Mac OS's root certificate store and our certificate provider. It is affecting a small number of connections currently but we believe the number will increase daily. We have set up a work-around, you will receive details via our Intercom system.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>07:25</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue. Connection failure are intermittent.</p>tag:status.foxpass.com,2005:Incident/18230282018-07-20T10:13:54-07:002018-07-30T13:41:32-07:00Issues With Delegated Authentication<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>10:13</var> PDT</small><br><strong>Resolved</strong> - A fix has been implemented and the problem has been resolved.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:51</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:14</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating issues with our delegated authentication mechanism</p>tag:status.foxpass.com,2005:Incident/17491852018-05-31T08:15:16-07:002018-05-31T08:15:16-07:00Radius Interruption<p><small>May <var data-var='date'>31</var>, <var data-var='time'>08:15</var> PDT</small><br><strong>Resolved</strong> - A new set of outbound IP addresses have been set up, and almost all customers have switched over. We are working to get more information from Okta about the block.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>06:27</var> PDT</small><br><strong>Monitoring</strong> - Okta is accepting traffic from the new IPs, we are continuing to monitor the situation.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>06:09</var> PDT</small><br><strong>Update</strong> - Our traffic to Okta's API is now coming from 52.5.197.216 and 35.153.22.124. You may need to update your Okta on-net whitelist. (please keep the existing ones active).</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>05:44</var> PDT</small><br><strong>Identified</strong> - We are unable to connect to Okta's API.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>05:43</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>05:34</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an interruption in our RADIUS service.</p>tag:status.foxpass.com,2005:Incident/17443742018-05-26T23:43:04-07:002018-05-26T23:43:04-07:00LDAP scheduled maintenance beginning<p><small>May <var data-var='date'>26</var>, <var data-var='time'>23:43</var> PDT</small><br><strong>Resolved</strong> - Maintenance has been completed.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>22:09</var> PDT</small><br><strong>Investigating</strong> - We are performing maintenance on the LDAP server.</p>tag:status.foxpass.com,2005:Incident/17030222018-04-24T09:33:19-07:002018-04-24T09:33:19-07:00Okta authentication failing for some customers<p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>09:33</var> PDT</small><br><strong>Resolved</strong> - Okta's incident has been resolved. See https://status.okta.com/.</p><p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>08:23</var> PDT</small><br><strong>Identified</strong> - Please follow Okta updates at https://status.okta.com/</p>tag:status.foxpass.com,2005:Incident/16325652018-03-07T19:47:45-08:002018-03-07T19:47:45-08:00Okta Incident<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>19:47</var> PST</small><br><strong>Resolved</strong> - Okta has resolved the issue.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>17:33</var> PST</small><br><strong>Identified</strong> - Okta is currently undergoing a service disruption. More information at https://status.okta.com/</p>tag:status.foxpass.com,2005:Incident/16226872018-02-28T14:13:39-08:002018-02-28T14:13:39-08:00Degraded RADIUS Performance<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>14:13</var> PST</small><br><strong>Resolved</strong> - The issue has been fixed.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>14:06</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.foxpass.com,2005:Incident/15807762018-01-27T23:46:44-08:002018-01-27T23:47:58-08:00Scheduled Maintenance<p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>23:46</var> PST</small><br><strong>Resolved</strong> - Maintenance completed</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>22:28</var> PST</small><br><strong>Monitoring</strong> - Maintenance beginning</p>tag:status.foxpass.com,2005:Incident/15286142017-12-06T23:21:11-08:002017-12-06T23:21:11-08:00Empty groups from sync'd sources<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>23:21</var> PST</small><br><strong>Resolved</strong> - Groups sync'd from external sources (Google, Azure, Okta, etc) may have been temporarily empty starting at approx. 5:20UTC. Most groups recovered within 30 minutes. All group memberships were fully restored by 7:00UTC.</p>