Making dynamic pages searchable

They have: 3 posts

Joined: May 2012

I have my product Actwitty - The Complete You.
This creates a social bio with topical sense of an individual based on their Twitter and Facebook.

The pages are created dynamically for users and they are available through their names, we are addressing them dynamically. I want to make all the pages crawlable for search engines. But at the same time I do not want to dump them in a directory which makes it accessible to all.

Please suggest which could be the best way to get this done.

Regards,
Sudhanshu

Greg K's picture

He has: 2,144 posts

Joined: Nov 2003

Have a dynamically generated sitemap.xml and submit that to google.

However, keep in mind, you say you don't want into a directory for it to be accessible by all, well that is exactly what you are doing by submitting it to any search engine.

-Greg

They have: 3 posts

Joined: May 2012

Thanks for the comments.
So as I understand competitors can be avoided only by obfuscating the sitemap.xml into multiple files, so that only a crawl + a good bit of scripting can help them know the details of links and corresponding priorities ?

Actually we have user pages and we do not want to "leakout in a easiest possible way" the list of users.
Is it better to have sitemap.xml or to have all links being pointed to by one or the other page ?

Thanks for the help.

-Sudhanshu

Greg K's picture

He has: 2,144 posts

Joined: Nov 2003

Again, if the pages are going to be public for a search engine to find them ANYONE can find them.

It doesn't take that much of scripting to find all the links on your site, software like Xenu will list all pages that can be found by just following links. Also, once you DO get all the pages you don't want your competition to find indexed by google, well all I will have to do is use google to see what all pages are indexed on your site.

So pretty much if you have user listing/pages you don't want found by others, then you also need to hide them from the search engines.

-Greg

They have: 3 posts

Joined: May 2012

Greg,

Thanks for a quick response. Point taken, my requirement is definitely to make key word and category based searchable bios of users based on their social media data.

I was just curious if there were ways, but you are right.
I agree its not difficult to list all links in a site and there are tools Xenu, Swiftype.

-Sudhanshu

aruldave's picture

They have: 12 posts

Joined: Jun 2012

Use GET method in URL to make your dynamic pages crawlable. Make sure you have provided the clear site map to increase the visibility of your web pages to search engine crawlers.

Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.