Communication Problem

There is a s2s communication problem between the XMPP cluster and other servers.

Update 16:10: solved by XMPP server restart.

New Primary Database

We got a new even more powerful primary database. The hardware patched old primary is now first secondary. Second secondary will also be around.

Operation resumed

The hardware has been patched up and is running again.

Hardware Failure

The current problem seems to be a hardware error on the main server and another hardware problem on the secondary.

We are aware, that this is virtually impossible. Nevertheless, the primary failed due to a hardware error and the secondary did not take over because of a different hardware related problem.

The hardware on the main server has been changed. Recover is under way. We expect that one of the DB servers will resume operation soon.

Website and client login down

There is a serious database problem. The failover did not come up automatically. We are working to recover.

Database Issues

The database suffers under high load of many client connections.

We are trying to reduce the load by disabling services temporarily. Buddylist and others may be affected.

Update 11:00: still optimizing components. The situation improves gradually, but will nevertheless take time.

Update 13:00: Operation resumed, but some services disabled. Buddylist status updates, points for the weekend (big sorry), Toplist, and some less visible components. Most important: chat works and people can meet each other. The world is back online.

Database down for Maintenance

The DB server will be locked for some time (expected: 1h) for maintenance. The websites will be affected and no new logins possible. Users who stay logged in and do not navigate will be able to continue their chats.

Update@02:12: Back online.

DB Maintenance

Portal very slow due to additional out of order DB backup.

XMPP Server Reboot

Server reboot to apply new database server config.

About the XMPP Authentication Refused Problem

Obviously the improved DB connection did not help as expected. There was a 2 hour outage on one of the cluster nodes.

This is a growth problem as not only server load grows, but also the effective coupling of sub systems by way of their increasingly loaded interfaces. Events once isolated begin to propagate between sub systems.

Investigation is under way. In addition, an alternate solution will be implemented today.

XMPP Server Reboot

Reboot to add improved DB connection module. Lets see if this makes the connections more stable.

XMPP Authentication Refused

One of the XMPP servers refuses to authenticate clients. DB connection problem.

Restarted the XMPP server. A solution is in the works. The client release soon to come will also be part of the solution.

Release Overload

A portal software release results in unexpected high load. Please be patient and try not to overload the web site.

Update: normal operation restored.

Update: Topcloud intentionally offline

XMPP Authentication Refused

One of the XMPP servers refuses to authenticate clients. DB connection problem.

(Restarting XMPP server + Taking actions to prevent the annoying client dialog box in the upcoming version. Improving DB connection.)

Update: Operation restored. Clients reconnect.