Use GET method in URL to make

aruldave — Wed, 20 Jun 2012 13:07:37 +0000

Use GET method in URL to make your dynamic pages crawlable. Make sure you have provided the clear site map to increase the visibility of your web pages to search engine crawlers.

Greg, Thanks for a quick

sudhanshu — Wed, 30 May 2012 12:19:21 +0000

Greg,

Thanks for a quick response. Point taken, my requirement is definitely to make key word and category based searchable bios of users based on their social media data.

I was just curious if there were ways, but you are right.
I agree its not difficult to list all links in a site and there are tools Xenu, Swiftype.

-Sudhanshu

Again, if the pages are going

Greg K — Wed, 30 May 2012 11:54:26 +0000

Again, if the pages are going to be public for a search engine to find them ANYONE can find them.

It doesn't take that much of scripting to find all the links on your site, software like Xenu will list all pages that can be found by just following links. Also, once you DO get all the pages you don't want your competition to find indexed by google, well all I will have to do is use google to see what all pages are indexed on your site.

So pretty much if you have user listing/pages you don't want found by others, then you also need to hide them from the search engines.

-Greg

Thanks for the comments. So

sudhanshu — Wed, 30 May 2012 11:45:17 +0000

Thanks for the comments.
So as I understand competitors can be avoided only by obfuscating the sitemap.xml into multiple files, so that only a crawl + a good bit of scripting can help them know the details of links and corresponding priorities ?

Actually we have user pages and we do not want to "leakout in a easiest possible way" the list of users.
Is it better to have sitemap.xml or to have all links being pointed to by one or the other page ?

Thanks for the help.

-Sudhanshu

Have a dynamically generated

Greg K — Mon, 28 May 2012 08:54:30 +0000

Have a dynamically generated sitemap.xml and submit that to google.

However, keep in mind, you say you don't want into a directory for it to be accessible by all, well that is exactly what you are doing by submitting it to any search engine.

-Greg