« Service Temporarily Unavailable | Main | Ramblings update: mod_security+Wordpress+cUrl »

Check Technorati Favorites update: learn php

By maurizio | May 8, 2007

I am really unlucky. When I first posted about this tool, Technorati just changed his internal layout so I had to rewrite some part of my code. Now for some reason is so slow that I had to remove part of it. You can’t simply click on a website to add it to your favorites, you need click to the technorati’s user page and then add whatever site you see on the list.
The reason is slow is very simple. This tool is nothing more than an automated browser. I browse your page and then check every users link and browse those pages too. So if you have 200 sites on your favorites and 1000 people on your site’s “favorited by”, it will take ages to read all of them, and will probably timeout before it finishes.

Reading pages this way is one of the most common way to get information automatically from the web. Most spammer and MFA sites creator use this kind of technique.It’s not that difficult. If you have some good ideas you will find that it’s very funny to create tools that read websites. The core of this kind of script is cURL. Here is an example taken from my tool:

function getPage($url){
$ch = curl_init();
$timeout = 5; // set to zero for no timeout
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$file_contents = curl_XXXX($ch); //change XXXX with exec !!!!
curl_close($ch);
$lines = array();
$lines = explode("\n", $file_contents);
//end curl
// Loop through our array, show HTML source as HTML source; and line numbers too.
foreach ($lines as $line_num => $line) {
$buffer .= $line;
}
return $buffer;
}
Please note: you should write exec instead of XXXX at line 7 (curl_XXXX)! I can’t do that myself or Wordpress will crash :-)
You don’t really need to be a PHP guru to understand this code. Actually you can easily test it if you have access to a web server with php support. You just need to create a file ( test.php ), add that piece of code and
$result = getPage("blog.nafurai.com");
echo $result;

At the beginning of the file you must add < ?php to let the server understand that you are writing php code. For the same reason you should put ? > at the end.
Then you should simply open that url with your browser and you will see blog.nafurai.com opening.

The code you just wrote is simple. The first big chunk is a function that open a webpage and store everything inside a big string, wich is basically a big object full of characters. With the second piece of code, you “ask” the program to read an URL (blog.nafurai.com) with the previous function and store it inside $result. The last line simply ask the program to show the content of the object $result;
Simple, isn’t it? You can easily imagine that printing the content is not that useful. Next time I’ll explain what you can do with that content.

Share This

Topics: Content Creation, Programming | 2 Comments »

Read other related posts:

2 Responses to “Check Technorati Favorites update: learn php”

1. Paula Neal Mooney Says:
   May 8th, 2007 at 9:42 am

   Hey Maurizio -

   Yeah, I still hadn’t run the Technorati tool yet…I’ve been so busy writing and promoting.

   And what exactly is PHP? I keep hearing a lot of talk about that.

   Anyway, would you do me a favor and Digg this:
   PAULA’S LIST OF BLOGGER SALARIES…ARE YOU ON THE LIST?

   Thanks!
   Paula

2. start programming with PHP | Nafurai Says:
   May 14th, 2007 at 4:10 am

   [...] main piece of code. You should do that if you use some piece of code and use it several times. On myTechnorati Favorites Tool explanation I show a function called getPage($url). You can put this function on the code above so it gets [...]

Comments