Comments on: Distributed Web Crawling with Tornado and Gearman http://www.iacquire.com/blog/distributed-web-crawling-with-tornado-and-gearman/?utm_source=rss&utm_medium=rss&utm_campaign=distributed-web-crawling-with-tornado-and-gearman Search is Our CRAFT Thu, 23 May 2013 10:21:00 +0000 hourly 1 http://wordpress.org/?v=3.5.1 By: Jon Hines http://www.iacquire.com/blog/distributed-web-crawling-with-tornado-and-gearman/#comment-514 Jon Hines Tue, 15 May 2012 20:37:00 +0000 http://blog.iacquire.com/?p=411#comment-514 thanks for the response Jeff!

]]> By: Jeff Nappi http://www.iacquire.com/blog/distributed-web-crawling-with-tornado-and-gearman/#comment-513 Jeff Nappi Sun, 13 May 2012 19:08:00 +0000 http://blog.iacquire.com/?p=411#comment-513 Hi Jon, sorry for the delayed reply. I plan on providing an easy to use Amazon Machine Image with a web based interface to this in the future, but for now yes you only need a single Ubuntu instance to run this. One way to do this is to sign up for Amazon Web Services and start by launching this AMI: https://console.aws.amazon.com/ec2/home?region=us-east-1#launchAmi=ami-a29943cb

Once you have everything set up by following the directions in the post, you would supply a list of URLs in a text file (one on each line) and pass the name of the text file to the TweetHandler.py script with the –url_file= parameter.

Feel free to contact me via e-mail if you have more questions – jeff _a_ iacquire.com

]]> By: Jon Hines http://www.iacquire.com/blog/distributed-web-crawling-with-tornado-and-gearman/#comment-512 Jon Hines Thu, 10 May 2012 14:36:00 +0000 http://blog.iacquire.com/?p=411#comment-512 so if i’m just using this on just 1 terminal, the setup above will be suffice? so if i have a list of url’s, how do I input this exactly?

]]> By: Jeff Nappi http://www.iacquire.com/blog/distributed-web-crawling-with-tornado-and-gearman/#comment-501 Jeff Nappi Wed, 02 May 2012 19:12:00 +0000 http://blog.iacquire.com/?p=411#comment-501 Something quite important I didn’t address in the post – the tests shown here are in fact only running on a single node. In order to run it in a distributed manner one would just launch TweetScout workers on additional nodes with the –jobserver=master-server-ip:4730 parameter. I will be making a follow-up post about this in the coming weeks.

]]>