As you are aware, our Zimbra e-mail and calendaring system experienced problems several times in the last week that resulted in outages. A combination of technical problems caused the system to go down and unfortunately, there is no easy fix to resolve them. I write to alert you to what we expect today and to answer questions you may have.
First, please be aware that it is likely the system will continue to be unstable today (Nov. 9) and that we may experience an additional outage. If we have an outage, it will take our staff roughly an hour to restore service. Tonight, we hope to install a “patch” that we believe will stabilize our system. Based on the recommendation by Zimbra, we have disabled the Instant Messaging capability until stability and performance improve.
Be assured that our ITS staff and the Zimbra corporate technicians assigned to our case are aware of the impact these outages cause to users, and that we are working around the clock to resolve them.
In anticipation of questions you may have, I have prepared some information in Question-Answer format to provide you with information. Please feel free to call the help desk at 278-5000 with any other questions you may have.
Q. What is causing Zimbra e-mail and calendar to go down? Why did it happen so many times last week?
Our system went down because of bugs and undisclosed changes by Zimbra in the new version we installed on October 28. Until these issues are resolved, Zimbra may continue to be unstable during peak demand hours.
Q. What is ITS doing to fix the problem(s)?
ITS staffers have been working closely with Zimbra engineers to resolve the bug as quickly as possible. Our normal response in such instances is to revert to the stable version we had prior to October 28, but the undisclosed change in Zimbra makes this impossible. Our campus situation is being monitored at the Vice President level at Yahoo (owners of Zimbra) and they have six engineers and a director of customer service dedicated to our case. Fresno State ITS staff members have been working nights and weekends to get this problem resolved as quickly as possible.
Q. How long will it take get this problem permanently fixed?
The Zimbra engineers have identified the bug causing our current issues and have created a patch. The patch is currently being tested and we expect to install it during a maintenance period on Monday night. It’s our hope and expectation that this patch will prevent future occurrences of the failures that caused our outages last week.
Q. It seems that things got worse after the recent “upgrade.” Was that a factor?
Yes. While there were some issues with our Zimbra system before the upgrade, the significant problems that we have experienced since October 28 are the result of problems associated with the upgraded version of Zimbra. Two issues were resolved by November 2. The continuing problems are the result of the bug, discussed above.
Q. Are there other “bugs” in Zimbra that will cause other problems for us?
As with any software, there will always be “bugs.” However, major bugs, like the one we have been experiencing, are rare. The amount of testing done by Zimbra and the fact that hundreds of other institutions are using their software mean that the likelihood of an additional major bug impacting our campus is unlikely. We do expect minor bugs, which is one of the reasons for upgrading or patching a system like Zimbra. One known minor bug that we are working with Zimbra to address is the fact that those with shared access to an address book may see multiple copies of each contact in that address book.
Q. Will the disruptions cause me to lose any of my e-mail or calendar entries?
The problems that we are experiencing have not resulted in the loss of e-mail or calendar entries and we do not expect data loss. The one exception to this is when a user has just begun writing a new message, or scheduling a meeting, at the time of a system outage. If a message or calendar entry was not saved before the outage, it would need to be recreated.
Q. Is there anything users can do to mitigate the problem?
Until we have verified that the problem has been corrected, users should log out of Zimbra and close their Web browser when they do not need to access their e-mail or calendar information. For example, while many of us currently leave Zimbra logged in throughout the day, logging out of Zimbra and closing your Web browser when you leave your office for meetings, lunch or other activities will help reduce the load on the system. Once you have logged out of Zimbra and closed your Web browser, you may re-launch your browser to access any other Web pages you need to use.
Please note: The busiest times on the Zimbra system are from 9-10 a.m, 1-2 p.m. and 4-5 p.m.
Q. Will Fresno State keep Zimbra as our e-mail and calendar provider or move to another system?
While the recent experience with an upgrade to the Zimbra system has caused serious problems for many, implementing a different e-mail and calendaring system takes many months of planning. Before making a decision of this magnitude, we would evaluate the use of Zimbra under more stable conditions.