andilinks — Sun, 21 May 2006 23:41:21 +0000

Quote: I have never used a "bot" before, but I have written my own program to harvest data from sites before.

I'd call that a bot, some people call them spiders but yes, any data mining program can be foiled by inconsistant formatting. Increasingly commercial bots identified by user-agent other than a vanilla browser are being blocked by savvy sysadmins. Was your program perl, php?

I have a commercial scraper that just grabs what I need. In this particular instance I'd buy the $30 database but if it were a $3,000 database I think I'd build a bot. It's a skill I'd like to develop more.

Greg K — Sun, 21 May 2006 22:58:50 +0000

The big thing about using a bot on the site given:

One state will list cities and townshops separately,another has them all in one list.
One state will have them in a table, one will have them in a list.
One state will list something like {CITY}, {COUNTY} one will have {CITY}, {STATE}

Lots of inconsistance between the states. I was going to be nice and gather them all myself and put a zipped up copy here, but after about the 4th state, was too much inconsistancy to be worth it.

I have never used a "bot" before, but I have written my own program to harvest data from sites before. I did it nicely, I figured out how fast I could manually follow every link, copy the data and paste it in excell. I took that time, doubled it and set my program to have a delay between page grabs. I let my program run during the night to harvest several thousand lists of contrators from a site for contractors to advertise. I figure, if I can do it by hand faster, they shouldn't be able to complain. Was tempted to write them afterwards and sugget how to deter people from doing what I did LOL.

-Greg

andilinks — Sun, 21 May 2006 20:32:23 +0000

Quote: Think of it this way, how long will it take you to pull all of that listing off of the site, etc, how much you think your time is worth per hour, how many hours until it is just cheaper to pay $30 for a database?

That's what bots are for... I don't use them often because they put such a load on the server and if I want a lot of data I'll do it in short spurts late at night to be kind to the site. But the way my site gets constantly hammered by bots I don't feel guilty about actually harvesting data this way.

In this instance though the $30 package may work just as well if you'd have to buy and learn a bot first. Sometimes you have to add a script to import directly to a database. Teleport Pro has an evaluation download.

Greg K — Sun, 21 May 2006 10:38:08 +0000

Good listing, but not so easy to just import into a database, have to click on every state individually to get full "City" listings, and then another link for the villiage, and another for townships, and so on for certain states.

(ie. check out Connecticut. Go to their full listing of cities. Then below all of that, click on a link for Fairfield County in the tbale below the lists of cities. There are some more cities not in the full list of "incorperated cities'.

Think of it this way, how long will it take you to pull all of that listing off of the site, etc, how much you think your time is worth per hour, how many hours until it is just cheaper to pay $30 for a database?

-Greg

ablaye — Sun, 21 May 2006 06:39:04 +0000

Well here is a list from answers.com

http://www.answers.com/topic/list-of-cities-in-the-united-states

You can add this to your DB.

cbc58 — Sat, 20 May 2006 22:08:27 +0000

Thank you.

Greg K — Sat, 20 May 2006 19:14:09 +0000

I don't know of any free ones, but I have in the past purchased a Zip Code datbase, you might want to do a search for that. This was a pretty good size database, so be prepared for that

I just looked it up, we bought our database from zipinfo.com We paid $30 for the one that includes area code and Alias cities. (I would recommend this one). In case you go looking up and wonder what the difference between "prefered city" and "alias city", take for example my location, 44224. The prefered city is Stow, Ohio, however Silver Lake, Ohio is also serviced by this, as well as parts of Cuyahoga Falls. (Cuyahoga falls does have it's own zip code, but Silver Lake doesn't)

-Greg

JeevesBond — Sat, 20 May 2006 19:07:55 +0000

Towns and cities for what country?