[teknoids] Issue with the CALI website on 8/21/16

Elmer Masters emasters at cali.org
Mon Aug 22 12:26:17 EDT 2016

Howdy folks.

Just wanted to let y'all know that we had an issue with the CALI
website yesterday for most of the day that resulted in some visitors
being told there were too many failed logins and their IP address was
banned. The problem was caused by our switch to a new load balancer
(LB) that was necessary for us to implement SSL across the main web
cluster. In a nutshell we missed updating a Drupal setting.

We successfully launched the new LB yesterday morning so now all main
CALI web traffic runs through https://www.cali.org/. As part of the
process we updated some Drupal module code and adjusted some of
Drupal's settings. We missed updating the reverse proxy setting so
Drupal saw all traffic as coming from the LB. This is bad because of
the flood control built in to Drupal 7 as security against attacks.

By some point in the early afternoon, as returning law students came
to the site, enough bad logins had accumulated (50 in an hour) for the
sole IP address being tracked that that one IP address was banned from
the site for 6 hours. Anyone arriving to the site after the ban went
into effect was not allowed to login without first requesting a
password reset, but as bad logins continued to accumulate even that
stopped working.

By the time Dan and I heard about the issue around 6 PM ET, it was
virtually impossible to login to the site. We quickly recognized the
problem, lifted the bans, and got to work sorting things out. We put a
temp fix in place overnight that kept the bans to a minimum and then
solved the issue by updating Drupal's settings this morning. By 9 AM
this morning everything was operating as it should.

We know that some students may not have been able to login and run
Lessons  or otherwise complete tasks on the site that were assigned to
them. We apologize for any disruption this issue caused.

Please remember that if you or your students and faculty have issues
with access to the CALI website or any problems using the site's
features the best way to get help is to send an email to
webmaster at cali.org.


Elmer R. Masters
Director of Technology
Center for Computer-Assisted Legal Instruction
emasters at cali.org    773-332-7508

More information about the Teknoids mailing list