Comments on: Help Needed for Web Scraping

By: Ismael Ghalimi

Ismael Ghalimi — Wed, 21 Mar 2007 21:09:26 +0000

Mike,

I will give it a shot!

-Ismael

By: Mike Parsons

Mike Parsons — Wed, 21 Mar 2007 20:14:15 +0000

Hi Ismael,

As a result of reading this blog entry and trying out some of the services mentioned in the comments, I decided to go ahead and create my own “simple” solution… You can read about it here.

By: Jack Pipe

Jack Pipe — Wed, 07 Mar 2007 19:45:35 +0000

Need web scraping? You name the website, we scrape the data for you.

By: Ismael Ghalimi

Ismael Ghalimi — Wed, 28 Feb 2007 23:44:28 +0000

Mike,

Thanks for sharing. I’ll take a look.

Best regards
-Ismael

By: Mike Parsons

Mike Parsons — Tue, 27 Feb 2007 14:37:17 +0000

Clipmarks looks interesting as well.

By: Ismael Ghalimi

Ismael Ghalimi — Thu, 22 Feb 2007 23:32:29 +0000

Stefan,

Thanks for the clarification. I’ll give it a try then.

Any plans to develop a version that would not require any software at all?

Best regards
-Ismael

By: Stefan Andreasen

Stefan Andreasen — Thu, 22 Feb 2007 23:16:49 +0000

OpenKapow only requires software to build the API’s. When built, everyone on the web can access them from the OpenKapow service, without any plugin or software. Thus, only you building it need to download the software, nobody else needs it. What you download from OpenKapow is basically a power web browser, with built-in visual scripting that allows you to build REST, JSON, RSS, and ATOM services, and deploy them on OpenKapow. Doing what you need should only take 15 minutes, after you ran over the OpenKapow tutorial and downloaded the software.

-Stefan

By: Ismael Ghalimi

Ismael Ghalimi — Thu, 22 Feb 2007 22:22:07 +0000

Cory,

That’s a great idea! I’ll give it a shot.

Best regards
-Ismael

By: Ismael Ghalimi

Ismael Ghalimi — Thu, 22 Feb 2007 22:21:37 +0000

Mike,

OpenKapow and screen-scraper are software based. Not an option.

I will give Ponyfish a shot though.

Best regards
-Ismael

By: Ismael Ghalimi

Ismael Ghalimi — Thu, 22 Feb 2007 22:20:47 +0000

Assaf,

Thanks for the tip. Looks good indeed.

Best regards
-Ismael

By: Ismael Ghalimi

Ismael Ghalimi — Thu, 22 Feb 2007 22:20:19 +0000

Nanek,

I could use Alexa’s web service indeed, but I need one for Google too.

Dapper looks pretty cool indeed. I’ll give it a shot.

OpenKapow seems to require the installation of software, so it’s not an option.

Best regards
-Ismael

By: Cory

Cory — Thu, 22 Feb 2007 14:07:50 +0000

Check out the ‘Content Analysis’ functionality in Yahoo! Pipes. This pipe that mashes up the New York Times front page with Flickr provides a good example.

By: Mike Parsons

Mike Parsons — Thu, 22 Feb 2007 11:25:05 +0000

Take a peek at one of these services: OpenKapow, Ponyfish, screen-scraper.

By: Ryan Armasu

Ryan Armasu — Thu, 22 Feb 2007 07:04:35 +0000

Take a look at scRUBYt! This is not necessarily the tool you may be looking for, but I found it pretty interesting nonetheless.

Congratulation on the new addition to the family.

-Ryan

By: Assaf

Assaf — Thu, 22 Feb 2007 06:18:37 +0000

I’m guessing you want something like Dapper, although it seems to be down now.

By: Nanek

Nanek — Thu, 22 Feb 2007 03:31:12 +0000

Well, for the Alexa information, most of it is available through the Alexa Web Information Service provided by Amazon. You would have to write a simple script that would pull this, and then output as JSON or RSS.

Alternatively, and more generically, you could use a service like Dapper that will walk you through a wizard and scrape information off of pages, then output it in many different formats. Its definitely worth a try, but I didn’t have any luck with the Alexa pages.

Another tool that is new is OpenKapow. I have not tried this one, but it looks promising based on the sample content out there.