The following are my opinions and recommendations only. They are not intended to be critical of the fine efforts of the people currently running the Forums, in fact having experienced similar "disaster" scenarios in the past I strongly empathize with the difficulties and pressures they were in. But drawing on my experiences, I have drawn up this list of items which are intended purely as <constructive> recommendations.
The vBulletin Plugin vulnerability appears to be likely a XSS exploit, if so then some basics on hardening against XSS
https://www.owasp.org/index.php/XSS_...on_Cheat_Sheet
Note the principles and methods described in the above link. Validation and scoping input is pretty basic and can(should have been) implemented at numerous places. vBulletin probably should be configurable to implement globally. Of course, the plugin creators (who apparently haven't existed since this last summer a few short months when the XSS vulnerability apparently was first published approx Sept) should have implemented. Probably the vBulletin modules which accept and parse imput (eg search boxes, create post) should be implementing.
In any case, I don't know what was behind the openSUSE forums' emergency response to press the panic button to implement a solution that wasn't staged so the whole website was taken down. There is no doubt that the consequences could have been serious if an actual attack was mounted.
So,
Here are my list of suggestions moving forward...
1. The response obviously was in the face of a threat that couldn't be managed(else I doubt anyone would have accepted the consequences of a website down for days). I strongly suggest implementing the solution I'd posted in my Wiki for a long time now - It has the ability to do incredibly powerful search and analysis of enormous amounts of data, even in realtime. In other words, no matter the load it should not be difficult to parse real time apache logs for malformed XSS strings typical of the SEO Plugin attack, and block those IP addresses automatically. You also would be able to do ad hoc analysis quickly on whether there is a pattern of mal-formed posts and from where.
http://en.opensuse.org/User:Tsu2/Ins...csearch-Kibana
2. Both before the incident and supporting future maintenance, I don't know if there is Policy and Guideline for both Disaster Response and Normal Maintenance. If not, policies should be created. It's normal for a business and should not be different for a community project. Included should be who to contact, possible flow diagrams for different kinds of scenarios and one should be created for just what happened while fresh in mind to improve future response. So, for instance when the software problem was discovered, was anyone in Network Administration brought in to provide input? Did anyone responsible for the software side even know that networking solutions existed that would likely have been sufficient to address the problem?
3. If don't already exist, resources should be pre-allocated for at least one (maybe many) staging platforms(my software development typically averages 3). Some should be completely isolated from the Production database, some should be connected to the Production database. Taking it a step further, full or nearly full implementations can be set in place for multiple uses, eg development, recovery. These additional platforms should be easy to allocate, they just require space and typically don't require significant resources but they do need to be secured. Of course, this requires virtualization which I don't know if SUSE has implemented (a year plus ago, it wasn't).
4. The following are recommendations which likely aren't currently implemented. The first (5) is a guideline on XSS. As I described above, XSS filtering can be implemented at numerous places, but if vBulletin isn't doing it there is little reason why it can't be deployed with relatively little effort on one's own. If filtering is global and to be applied to numerous servers, then it might be more practical to implement in a Proxy in front of the actual webservers. If the websites to be protected are few, then it can be implemented per server or application. In other words, it pays to communicate between network administration and application web admins. If co-ordination is spotty, then maybe "defense in depth" allows you to sleep easier (own your own solutions to problems).
After the XSS Prevention guide, I'm providing the links to the Apache Rewrite Module(6). As of today, my tests indicate that the old links <do not work>. I think that regardless whether a decision is made to deploy the new web root or go back to the old one, <support> for the old root is critical to avoid breaking all the links that point to old forum posts. Note that according to the Apache documentation, the mod_rewrite module is <not preferred> but I'm giving you the link anyway because in the short run implementing should only require 1 to 2 lines of code.
5. The XSS Prevention Guide (from OWASP)
https://www.owasp.org/index.php/XSS_...on_Cheat_Sheet
6. Some Apache mod_rewrite documentation, likely specific to IMO what the Forums needs to implement
The IBM documentation for mod_rewrite, good for fast and simple but lacking depth
http://publib.boulder.ibm.com/httpse...riteguide.html
The Apache documentation for mod_rewrite
http://httpd.apache.org/docs/2.2/rewrite/
If re-write is still used, likely preferred config
http://httpd.apache.org/docs/2.2/rewrite/vhosts.html
Alternatives to simple URL re-write
http://httpd.apache.org/docs/2.2/rewrite/avoid.html
Preferred solution probably
http://httpd.apache.org/docs/2.2/vhosts/mass.html
Repeating myself a bit if it wasn't clear, any implementation of both XSS hardening and web root re-write can and should be implemented on a staging platform before implementing in Production. Properly done, switching over should be done transparently to all Users (although selecting a very low traffic time is highly advised).
TSU
The vBulletin Plugin vulnerability appears to be likely a XSS exploit, if so then some basics on hardening against XSS
https://www.owasp.org/index.php/XSS_...on_Cheat_Sheet
Note the principles and methods described in the above link. Validation and scoping input is pretty basic and can(should have been) implemented at numerous places. vBulletin probably should be configurable to implement globally. Of course, the plugin creators (who apparently haven't existed since this last summer a few short months when the XSS vulnerability apparently was first published approx Sept) should have implemented. Probably the vBulletin modules which accept and parse imput (eg search boxes, create post) should be implementing.
In any case, I don't know what was behind the openSUSE forums' emergency response to press the panic button to implement a solution that wasn't staged so the whole website was taken down. There is no doubt that the consequences could have been serious if an actual attack was mounted.
So,
Here are my list of suggestions moving forward...
1. The response obviously was in the face of a threat that couldn't be managed(else I doubt anyone would have accepted the consequences of a website down for days). I strongly suggest implementing the solution I'd posted in my Wiki for a long time now - It has the ability to do incredibly powerful search and analysis of enormous amounts of data, even in realtime. In other words, no matter the load it should not be difficult to parse real time apache logs for malformed XSS strings typical of the SEO Plugin attack, and block those IP addresses automatically. You also would be able to do ad hoc analysis quickly on whether there is a pattern of mal-formed posts and from where.
http://en.opensuse.org/User:Tsu2/Ins...csearch-Kibana
2. Both before the incident and supporting future maintenance, I don't know if there is Policy and Guideline for both Disaster Response and Normal Maintenance. If not, policies should be created. It's normal for a business and should not be different for a community project. Included should be who to contact, possible flow diagrams for different kinds of scenarios and one should be created for just what happened while fresh in mind to improve future response. So, for instance when the software problem was discovered, was anyone in Network Administration brought in to provide input? Did anyone responsible for the software side even know that networking solutions existed that would likely have been sufficient to address the problem?
3. If don't already exist, resources should be pre-allocated for at least one (maybe many) staging platforms(my software development typically averages 3). Some should be completely isolated from the Production database, some should be connected to the Production database. Taking it a step further, full or nearly full implementations can be set in place for multiple uses, eg development, recovery. These additional platforms should be easy to allocate, they just require space and typically don't require significant resources but they do need to be secured. Of course, this requires virtualization which I don't know if SUSE has implemented (a year plus ago, it wasn't).
4. The following are recommendations which likely aren't currently implemented. The first (5) is a guideline on XSS. As I described above, XSS filtering can be implemented at numerous places, but if vBulletin isn't doing it there is little reason why it can't be deployed with relatively little effort on one's own. If filtering is global and to be applied to numerous servers, then it might be more practical to implement in a Proxy in front of the actual webservers. If the websites to be protected are few, then it can be implemented per server or application. In other words, it pays to communicate between network administration and application web admins. If co-ordination is spotty, then maybe "defense in depth" allows you to sleep easier (own your own solutions to problems).
After the XSS Prevention guide, I'm providing the links to the Apache Rewrite Module(6). As of today, my tests indicate that the old links <do not work>. I think that regardless whether a decision is made to deploy the new web root or go back to the old one, <support> for the old root is critical to avoid breaking all the links that point to old forum posts. Note that according to the Apache documentation, the mod_rewrite module is <not preferred> but I'm giving you the link anyway because in the short run implementing should only require 1 to 2 lines of code.
5. The XSS Prevention Guide (from OWASP)
https://www.owasp.org/index.php/XSS_...on_Cheat_Sheet
6. Some Apache mod_rewrite documentation, likely specific to IMO what the Forums needs to implement
The IBM documentation for mod_rewrite, good for fast and simple but lacking depth
http://publib.boulder.ibm.com/httpse...riteguide.html
The Apache documentation for mod_rewrite
http://httpd.apache.org/docs/2.2/rewrite/
If re-write is still used, likely preferred config
http://httpd.apache.org/docs/2.2/rewrite/vhosts.html
Alternatives to simple URL re-write
http://httpd.apache.org/docs/2.2/rewrite/avoid.html
Preferred solution probably
http://httpd.apache.org/docs/2.2/vhosts/mass.html
Repeating myself a bit if it wasn't clear, any implementation of both XSS hardening and web root re-write can and should be implemented on a staging platform before implementing in Production. Properly done, switching over should be done transparently to all Users (although selecting a very low traffic time is highly advised).
TSU