SureVoIP Hosted system crash - Wednesday 21st November 2012 at 16:50 and 17:00

At 16:50 today we experienced a system crash on our hosted platform. It took down live calls, including my own. It was caused by a feature code that was enabled in the summer but not really in use. It was fully tested on our test platform but some strange combination of digits caused what is known as a "core dump". The software we use (FreeSWITCH) does not normally experience this and we have only seen it a few times over the past 3 years. We are fully reviewing our new feature rollouts and management procedures for this.

We have disabled this feature code and moved it back to testing until we identify the full cause. 

The hosted service was restarted as soon as was detected, which was before our automated monitoring. Because it was a software crash and was restarted so close to the crash, the failover system did not take over as it is designed for hardware failure and longer software failure outages, i.e. greater than 5 minutes.

The second crash at 17:00 was caused by this feature code being used again before we had disabled it completely. After the past few weeks this is the last thing that I wanted to happen.

We'll keep the SureVoIP Hosted service marked as a cross on the Service Status page for a while in case our investigations have missed anything, but we are confident that they have not.

If you are having registration issues then as advised to those that have called and emailed support, please reboot your phones. We are however seeing everyone's handsets fully connected now.

Should any one wish to have a refund for this months SureVoIP Hosted rental because of this 2 minute outage, then please contact us.

Thanks and sorry,

Gavin.