SEO Security Advisory - Check your sites NOW

Rob · February 16th, 2007, 06:20 PM

This is an all points SEO Advisory - Check your websites now!

Many websites are wide open to abuse - competitors can with little effort affect YOUR rankings.

As most webmasters, and good SEO's know, google frowns on duplicate content and many sites have an all too often overlooked security hole (maybe yours) which would allow anyone to create duplicate content WITHIN YOUR OWN site, which could affect your page's rankings.

The best way to show the security hole in action, is by direct example
Take a look at this page, and the URL - VERY CLOSELY.

http://communityseo.com/forums/Commu...-List-t35.html

Now look here
http://communityseo.com/forums/this-...icate-t35.html
(humble apologies to the site in question here - if you read this, nothing personal)

What you have witnessed here, is that I have just created a crawlable link to a duplicate content page - which the search engines will go along an index - I can create hundreds of these all pointing to the same page which may adversely affect the ranking of that page.

To be fair, you can create duplicate content in this way on most sites by appending querystring values although this isn't anywhere near as bad as being able to inject new, static urls as shown above.

Many applications are prone to this security hole, including many implementations of static URL's on wordpress, ipb, phpbb, joomla, mambo - in fact from my extensive research, this list goes on and on.

Truly static sites, of the true .htm kind, are not prone to this hole, however websites that use mod-rewrite or some other form of url rewriting in 90% of cases have holes in them.

Lucky people with vBulletin, running the vBSEO add-on are almost entirely protected from this security hole.

Make no mistake about this... this IS a security hole - a security hole is defined by the existence of some hidden method, whereby somebody up to no good can inflict damage - the damage in the case of this security hole would be damage to page rankings.

What can be done then?

Well, from last week when I discovered this, I have started to make new websites differently. All pages on new websites I make that use url-rewriting I have made to be self-aware of their own url. If these pages get called using an unexpected URL, it is detected and a 301 redirect occurs to the real URL - viola... protection - easy!

Well, that just about concludes this post and I truly wish you have found this helpful. I hope that this knowledge share doesn't have you all panicking like mad, but it's definitely worth thinking about - not all competitors will go out and start screwing people over... but some will - they have for years, and will continue to do so with any other new method that gets uncovered.

If you have found your website to be vulnerable to this and want help implementing protection for your website, then please let me know via private message on this website.

If you have been helped, or had your bacon saved, then please consider to make a donation, as it helps this site cope with it's huge bandwidth costs.

JacobHaug · February 16th, 2007, 07:09 PM

WOW, and in short...that is totally why we have our SEO Guru...lol

sannbe · February 17th, 2007, 01:39 AM

Bless you oh great SEO guru. We bow to your great knowledge and wisdom. Besides you are always a great help.

spinal007 · February 17th, 2007, 10:38 AM

Very good post and definitely something all developers and SEOs will have to be wary of.

The worrying part is how many people out there (specially SEOs) are using URL re-writes and other techniques without giving any consideration to the security issues involved. They sure are a huge part of SEO but this post really shows we are playing with fire.

I knew about the vulnerability and I thought my CMS was secure, but after a closer look I have quite a list issues to deal with.

I better get started...

Rob · February 17th, 2007, 12:17 PM

When I started looking into the extent of this on one of my sites, it dawned on me how potentially serious this could be. Upon further research and investigation, my findings were alarming. Further research has me wishing I'd said 'millions' of sites were affected - as tens of thousands is way, way underestimated. I have seen some HUGE corporate sites with this hole.

This security hole also leaves me thinking that the recently patched part of google's algorithm which stops googlebombing, could not possibly have taken this into account - I think google bombing is still possible on these sites and to potentially damaging proportions.

"googlebomb' was the practice by which it was possible for loads of bloggers to get george bush to the top of google for 'miserable failure'

JacobHaug · February 19th, 2007, 02:15 PM

hahaha...that is horrible, I have not heard of that...lol

moshe · February 19th, 2007, 02:40 PM

Who was george bush?

Mike

JacobHaug · February 19th, 2007, 02:42 PM

George Bush is the president in the USA...lol

February 19th, 2007, 02:46 PM

Hey-

Im one of the admins on communitySEO and just wanted to say that when users install our script, they have the option to either allow for loose filtering (which is what you described) or strict, with suggest.
So if you went to forum-help-topic-22.html it wouldnt take you to forums-help-topic-22.html but instead ask you if that is where you meant to go (or 301 you there).

You can view the full feature set of community seo for IPB @ http://communityseo.com/ipb_seo.htm

I would also like to say, that on the majority of pages (especially forums) you will never get a duplicate content punishment, since all pages will have differences. Google only really brings out the dupe punishment for 'near' copys.

For example:
When I went to http://communityseo.com/forums/Commu...-List-t35.html
I got this text @ the bottom:

4 User(s) are reading this topic (3 Guests and 0 Anonymous Users)
1 Members: Dan

0.4418 sec

3.37

12 queries

GZIP Enabled
Time is now: 19th February 2007 - 01:47 PM

Now for the second link:

http://communityseo.com/forums/this-...icate-t35.html

5 User(s) are reading this topic (4 Guests and 0 Anonymous Users)
1 Members: Dan

0.7951 sec

3.63

12 queries

GZIP Enabled
Time is now: 19th February 2007 - 01:48 PM

Now im not saying this will totally escape and duplicate content punishments, but google is smart enough to recognize it as there has been no issues for this across any sites using similar software that I am aware of.

Outstanding writeup, and great forum you guys have here.

Cheers

Rob · February 19th, 2007, 05:24 PM

Hi Dan....

First of all, let me thank you for stopping by here....

Believe me when I say that the choice of your site as an example was merely co-incidence and nothing personal - I hope you don't see it as an attack on your product - especially as a fix is easy.

That said however, I have to totally disagree with your comments here.....

why?
Because, I see evidence to the contrary on client sites every day??
- well, partially.

Because I've just swallowed an SEO handbook and live in the SEO world?? Actually.... no Dan - you rarely see me in SEO communities because I trust nothing but my own findings - too much poppycock in many SEO communities now - too many guess-merchants, if you know what I mean.

Actually, its not really any of those....

I work mostly in ridding sites of dupe content and taking measures to have dupes removed from indexes as fast as possible. I use 301 redirection mainly on small sites (takes longer) and also manual removal using google webmaster tools.

Once I can guard a site with unique urls and prevent dupes creeping in, I always (did I say ALWAYS?) see I noticeable jump in rankings.

Most forums are the WORST sites for SEO anyway, you would be hard pressed to identify a duplicate content penalty. But, what you can do, is take an established forum, annihilate duplicate content from indexes and take solid protection measures to ensure one url per resource..... only then will you realise you were being penalised.

However, it not just that easy is it... being able to inject querystrings into nearly every url on the web is not really that eyebrow raising.... search engines index pages with querystrings and I think you will find this is where your confusion creeps in here...

Duplicate content pages where the duplicate aspect is caused by querystring alone (as is the case with most forums) almost definitely carries a far lesser dupe content penalty. Search engines are well aware of the fact that querystrings control functionality on a site and the querystring combinations for any one page are sometimes very numerous.

I truly believe however (and it's a very reasonable and likely assumption) the same principles do not apply to static pages. Products including your own, that use url rewriting should enforce URL integrity for the sake of your user base.

I created one dupe content URL above.... that's nothing.... but imagine there were several hundred for many pages on your site, all linked to around several tens of sites on the net.... I would be interested to see the effects of that, and whether the issue may perhaps be taken a little more seriously by application vendors.

One other aspect you may wish to look at is the humanly readable URL's such as mysite.com/books-9/mary-had-a-lamb-10 .... yes, thats an advantage of static over dynamic --- pretty urls. The potential exists however for someone to link to you in a high traffic environment where a lot of people will read the url - ok, it's not that bad - but could be embarassing more than damaging.

I think the worst aspect however is the following senario...

http://communityseo.com/forums/Commu...ional-t35.html

To explain the above.... yes... it's another duplicate link.... but it's more than just that - it's a negative statement.

Google likes keywords in urls doesnt it? Google also likes those keywords to be on the page... in this case, they are (go ahead and look). Now, cue up a few hundred sites to link to this and I kind of feel the googlebomb may have just been re-invented........ or not, but hey... the page exists, it's on your site... the keywords are in the url... and they're on your page, so I personally would feel a tad uncomfortable there.

Again forgive me for this, but I have to demonstrate what i mean in order for this to be looked at with the severity it deserves. (by the way, all outbound links here are rel="nofollow" )

I hope you perceive this in a different light now, and I am glad you have the option to enforce checking... although what possible purpose could it serve to be off? Seems an odd option to have.

February 19th, 2007, 05:59 PM

In regards to the duplicate content, I feel the slight difference upon every page load would keep it from being marked as duplicate, but before we get to far, i did want to mention that communityseo does have this feature already built in, where you can 301 badlinks to the correct one, or provide a page where the user can click through also. (or you can leave it off, as we are in this case)

I wasnt trying to say you were wrong, I just wanted to point out that communityseo does have this feature and it can easily be enabled. Its just a fine line though, as having more pages in google can also be a benefit if google does not mark them as duplicate content.

If you have a static based site without the time, query count, users reading a topic, etc.. Then its very likely you can indeed be hit with duplicate content, but I believe its difficult in a forum enviorment to have 2 urls be matched exactly due to the constantly changing variables. The duplicate content detector by google is moreso used as prevention for MFA (made for adsense) sites where there is no dynamic content, and it is easy for google to pick up dupes. There are some different situations as well though. Google penalizes sites more heavily in my experience if they use or rip wikipedia content or dmoz content often.

Anyways

I do agree with you that this is a potential issue, but I also feel that there is much worse things you can do to a site than this, and easier to do as well. (bad neighborhood linking, setting up a mod-rewrite on another domain to mirror content from another, etc..)

If you have any suggestions or ideas you would like to see in communityseo please let me know, as we are indeed trying to make the best product available, and want to offer things other software does not.

Cheers-

Dan

Rob · February 19th, 2007, 07:19 PM

Quote:

Originally Posted by bigdan

In regards to the duplicate content, I feel the slight difference upon every page load would keep it from being marked as duplicate,

not when the only thing making it 'ever so slightly' not duplicate is in the footer almost and accounts for around 1% - 3% of the content (on most pages)

Quote:

Originally Posted by bigdan

but before we get to far, i did want to mention that communityseo does have this feature already built in

That is important I feel, so kudos for your users.

Quote:

Originally Posted by bigdan

If you have a static based site without the time, query count, users reading a topic, etc.. Then its very likely you can indeed be hit with duplicate content, but I believe its difficult in a forum enviorment to have 2 urls be matched exactly due to the constantly changing variables.

Do you think duplicate filters are only applied when the page is an exact duplicate? If so, then I think you may be shocked to learn otherwise - again, the content you refer to is mostly in the last chunk of code google will read - All the most important seo factors (title, description, h1, opening paragraphs, blah, blah) will be identicala nd add to this that the entire page content as a whole will differ by between 1 and 3% on average and thats easy low enough for penalities to occur.

Quote:

Originally Posted by bigdan

The duplicate content detector by google is moreso used as prevention for MFA (made for adsense) sites where there is no dynamic content, and it is easy for google to pick up dupes. There are some different situations as well though. Google penalizes sites more heavily in my experience if they use or rip wikipedia content or dmoz content often.

I totally agree with that, cross site dupe content is pain in the neck of all google users.

Quote:

Originally Posted by bigdan

....I also feel that there is much worse things you can do to a site than this, and easier to do as well. (bad neighborhood linking, setting up a mod-rewrite on another domain to mirror content from another, etc..)

The issue at hand allows a third party to affect rankings whereas the bad neighbourhood linking you refer to would be entirely the fault of the webmaster as he is the only person who could link to a bad neighbourhood. We both know that the other way round (a bad neighbourhood linking to you) is a non penalty since you have no control over who links to you - I cant see how this is relevant.

Quote:

Originally Posted by bigdan

If you have any suggestions or ideas you would like to see in communityseo please let me know, as we are indeed trying to make the best product available, and want to offer things other software does not.

Well, my ideas would really be to eliminate all duplicate content (for instance, your last post links land on a dupe page - there may be others), and focus on the aspects of SEO you can control - there are many pointless items of information and links that spiders need not see - only members. There are also other aspects of on-page SEO such as a meta description, h1, h2, etc you may wish to look at.

Ok, I'm naming the obvious ones here, but they are the most basic requirement that all good seo'd pages should contain.

Good luck with your project Dan, and shout me when the project is further down the road and I'll take another peek.

February 19th, 2007, 09:05 PM

Good points, although my experience with the results differ from yours, so I have a different outcome. I feel that the "possible" duplicate content penaliziation is not outweighed by the benefit of having more pages listed in google. I know we differ on this, so I wont ask for you to try and disprove me, which is why our SEO system allows both methods.

Regardless, I activated the setting on communityseo.com, so you can see how it 301's now if you revist your first example.

-Cheers

Rob · February 19th, 2007, 10:34 PM

Quote:

Originally Posted by bigdan

I feel that the "possible" duplicate content penaliziation is not outweighed by the benefit of having more pages listed in google

I think the entire SEO world would disagree, but it's your users choice I guess :S.

I would much rather a 1 high ranking page, than 1 much lower ranked one and a few supplemental results.... but then, I'm funny like that.

abrewitt · May 15th, 2007, 06:25 AM

Thanks for heads up Rob! - I agree and am alos gald we have our SEO Guru here!

darehammer · September 20th, 2007, 11:29 AM

you guyz rock man !!!!!!

alexgeek · October 15th, 2007, 03:59 AM

I just figured out the problem! Didn't understand before.
If you have time, could you tell me how to 301 redirect wrong urls?

Rob · October 15th, 2007, 05:22 AM

Code:


			
<?
    Header( "HTTP/1.1 301 Moved Permanently" ); 
    Header( "Location: http://www.new-url.com" ); 
?>

Lucleonhart · November 9th, 2007, 08:50 AM

Hm.. just thought trying that here.. isn't it the same?
http://www.webforumz.com/webforumz-c...your-sites.htm
and
http://www.webforumz.com/webforumz-c...-dublicate.htm

??

Marc · November 9th, 2007, 01:16 PM

I dont think it is because when you click: http://www.webforumz.com/webforumz-c...-dublicate.htm it takes you to: http://www.webforumz.com/webforumz-c...your-sites.htm.

It must be using a URL Rewrite.