Converting site content into RSS feed
Screen scraping your way into RSS
By Dennis Pallett
Introduction
RSS is one the hottest technologies at the moment, and even big web publishers (such as the New York Times) are getting into RSS as well. However, there are still a lot of websites that do not have RSS feeds.
If you still want to be able to check those websites in your favourite aggregator, you need to create your own RSS feed for those websites. This can be done automatically with PHP, using a method called screen scrapping. Screen scrapping is usually frowned upon, as it's mostly used to steal content from other websites.
I personally believe that in this case, to automatically generate a RSS feed, screen scrapping is not a bad thing. Now, on to the code!
Getting the content
For this article, we'll use PHPit as an example, despite the fact that PHPit already has RSS feeds .
We'll want to generate a RSS feed from the content listed on the frontpage . The first step in screen scraping is getting the complete page. In PHP this can be done very easily, by using implode(file("", "[the url here]")); IF your web host allows it. If you can't use file() you'll have to use a different method of getting the page, e.g. using the CURL library .
Now that we have the content available, we can parse it for the content using some regular expressions. The key to screen scraping is looking for patterns that match the content, e.g. are all the content items wrapped in <div>'s or something else? If you can successfully discover a pattern, then you can use preg_match_all() to get all the content items.
For PHPit, the pattern that match the content is <div class="contentitem">[Content Here]<div>. You can verify this yourself by going to the main page of PHPit, and viewing the source.
Now that we have a match we can get all the content items. The next step is to retrieve the individual information, i.e. url, title, author, text. This can be done by using some more regular expression and str_replace() on the each content items.
By now we have the following code;
<?php
// Get page
$url = "http://www.phpit.net/";
$data = implode("", file($url));
// Get content items
preg_match_all ("/<div class=\"contentitem\">([^`]*?)<\/div>/", $data, $matches);
Like I said, the next step is to retrieve the individual information, but first let's make a beginning on our feed, by setting the appropriate header (text/xml) and printing the channel information, etc.
// Begin feed
header ("Content-Type: text/xml; charset=ISO-8859-1");
echo "<?xml version=\"1.0\" encoding=\"ISO-8859-1\" ?>\n";
?>
<rss version="2.0"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:admin="http://webns.net/mvcb/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<channel>
<title>PHPit Latest Content</title>
<description>The latest content from PHPit (http://www.phpit.net), screen scraped!</description>
<link>http://www.phpit.net</link>
<language>en-us</language>
<?
Now it's time to loop through the items, and print their RSS XML. We first loop through each item, and get all the information we get, by using more regular expressions and preg_match(). After that the RSS for the item is printed.
<?php
// Loop through each content item
foreach ($matches[0] as $match) {
// First, get title
preg_match ("/\">([^`]*?)<\/a><\/h3>/", $match, $temp);
$title = $temp['1'];
$title = strip_tags($title);
$title = trim($title);
// Second, get url
preg_match ("/<a href=\"([^`]*?)\">/", $match, $temp);
$url = $temp['1'];
$url = trim($url);
// Third, get text
preg_match ("/<p>([^`]*?)<span class=\"byline\">/", $match, $temp);
$text = $temp['1'];
$text = trim($text);
// Fourth, and finally, get author
preg_match ("/<span class=\"byline\">By ([^`]*?)<\/span>/", $match, $temp);
$author = $temp['1'];
$author = trim($author);
// Echo RSS XML
echo "<item>\n";
echo "\t\t\t<title>" . strip_tags($title) . "</title>\n";
echo "\t\t\t<link>http://www.phpit.net" . strip_tags($url) . "</link>\n";
echo "\t\t\t<description>" . strip_tags($text) . "</description>\n";
echo "\t\t\t<content:encoded>[CDATA[ \n";
echo $text . "\n";
echo " ]]</content:encoded>\n";
echo "\t\t\t<dc:creator>" . strip_tags($author) . "</dc:creator>\n";
echo "\t\t</item>\n";
}
?>
And finally, the RSS file is closed off.
</channel>
</rss>
That's all. If you put all the code together, like in the demo script, then you'll have a perfect RSS feed.
Conclusion
In this tutorial I have shown you how to create a RSS feed from a website that does not have a RSS feed themselves yet. Though the regular expression is different for each website, the principle is exactly the same.
One thing I should mention is that you shouldn't immediately screen scrape a website's content. E-mail them first about a RSS feed. Who knows, they might set one up themselves, and that would be even better.
Download sample script
About the Author
Dennis Pallett is a young tech writer, with much experience in ASP, PHP and other web technologies. He enjoys writing, and has written several articles and tutorials. To find more of his work, look at his websites at http://www.phpit.net , http://www.aspit.net and http://www.ezfaqs.com
Sending SMS Thru HTTP
By Farheen
There are an infinite number of reasons why you might want to use PHP to send SMS. You might want to add a "send by SMS" option to your headlines, you might want to provide 24/7 support in which your technican is alerted by SMS or you might want to provide your viewers with Free SMS to drive traffic to your site.
Although it is possible to send SMS via e-mail, which we will cover another time, this tutorial will focus on the use of HTTP methods "get" & "post". For those of us that many not know this, using HTTP basically means the use of forms, just like a contact form <form></form>, except that these will be submitted automatically as opposed to manually.
Although this tutorial can be used for any gateway that provides access via HTTP, it is based on TM4B's <a href="http://www.tm4b.com/">SMS Gateway</a> because a) they are the only gateway i know that have a 'simulation' mode for tweaking your scripts, b) they don't have any set-up fees and their prices are low, and c) they are reliable and i use them.
Step 0: Understanding the requirements of the gateway.
Full details about connecting to TM4B are provided on their <a href="http://www.tm4b.com/connectivity/">SMS API</a> page. They basically require us to provide six mandatory pieces of data:
i. username - our username
ii. password - our password
iii. msg - our SMS message
iv. to - the one or more recipients of our message
v. from - our sender id
vi. route - the route of the message (i.e. first class or business class)
And we will add a seventh, which is optional... sim - simulate.
They will be expecting us to send our messages to them via HTTP requests, similar this one:
http://www.tm4b.com/client/api/send.php?username=abcdef
&password=12345&msg=This+is+sample+message.
&to=447768254545%7C447956219273%7C447771514662
&from=MyCompany&route=frst?=yes"
which you can test by pasting it into your browser's address bar:
Step 1: Prepare our request
The first step is to save our data as variables and then convert them into a url request. There are different ways of doing this, but this is a very innovative and useful way:
Code:
<?php
$request = ""; //initialise the request variable
$param[username] = "abcdef"; //this is the username of our TM4B account
$param[password] = "12345"; //this is the password of our TM4B account
$param[msg] = "This is sample message."; //this is the message that we want to send
$param[to] = "447768254545|447956219273|447771514662"; //these are the recipients of the message
$param[from] = "MyCompany";//this is our sender if
$param[route] = "frst";//we want to send the message via first class
$param[sim] = "yes";//we are only simulating a broadcast
foreach($param as $key=>$val) //traverse through each member of the param array
{
$request.= $key."=".urlencode($val); //we have to urlencode the values
$request.= "&"; //append the ampersand (&) sign after each paramter/value pair
}
$request = substr($request, 0, strlen($request)-1); //remove the final ampersand (&) sign from the request
/*
This will produce the following request:
username=abcdef&password=12345&msg=This+is+sample+message.
&to=447768254545%7C447956219273%7C447771514662
&from=MyCompany&route=frst?=yes
?>
Step 2: Open up our connection with TM4B and send the request
In step 0, we saw that the request could be actioned by pasting it into the browser window. But what we really want is for this to take place behind the scenes.
The following 2 pieces of code do exactly that. They open up a connection with the gateway, send the SMS message(s) and collect their message ID's which are presented within the response header.
Method 1 : fosckopen method
Code:
<?php
//First prepare the info that relates to the connection
$host = "tm4b.com";//although you can use an ip address, it is easier to just use tm4b.com
$request_length = strlen($request);// when we post the header, we have to also include it's length
$script = "/client/api/send.php";
$method = "POST"; //Replace with "GET" if required.
if($method=="GET") $script .= "?$request";//Appends the request if "GET" is being used.
//Now comes the header which we are going to post. This is where our messages details will be sent over.
$header = "$method $script HTTP/1.1\\r\\n". //
"Host: $host\\r\\n".
"User-Agent: HTTP/1.1\\r\\n".
"Content-Type: application/x-www-form-urlencoded\\r\\n".
"Content-Length: $request_length\\r\\n".
"Connection: close\\r\\n\\r\\n".
"$request\\r\\n";
//Now we open up the connection
$socket = @fsockopen($host, 80, $errno, $errstr);
if ($socket) //if its open, then...
{
fputs($socket, $header); // send the details over
while(!feof($socket)) $output[] = fgets($socket); //get the results
fclose($socket);
}
//print "<pre>";print_r($output);print "</pre>";//the message id's will be kept in one of the $output values
?>
Whilst fsockopen may be more familar to most of us, it can only handle non-secure URL's. Furthermore, difficulty may be experienced when parsing responses for large requests as the responses are transferred in chunks.
Method 2 : Curl method
Whilst Curl might sound new, it is a very impressive library that allows you to connect and communicate to many different types of servers with many different types of protocols. You can find more info in the <a href="http://uk2.php.net/curl">PHP Manual</a>.
Code:
<?php
$url = "https://www.tm4b.com/client/api/send.php"; //although we have used https, you can also use http
$ch = curl_init(); //initialize curl handle
curl_setopt($ch, CURLOPT_URL, $url); //set the url
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); //return as a variable
curl_setopt($ch, CURLOPT_POST, 1); //set POST method
curl_setopt($ch, CURLOPT_POSTFIELDS, $request); //set the POST variables
$response = curl_exec($ch); //run the whole process and return the response
curl_close($ch); //close the curl handle
//print $response; //show the result onscreen for debugging
?>
Step 3: That's It
Although it took me a long time to find Curl, i think it is the best, neatest and quickest option (assuming your version of PHP supports it) as it can send thousands of messages in one go, gives no problems in parsing the message ID's and uses either a secure or non-secure url.
The above took me ages, and i hope it saves you time.
About the Author
Farheen is a text-messaging enthusiast. She has done a lot of research in the world of SMS and maintains a popular blog called BestKeptSimple. PHP Validators & Client-side Validation
By Dennis Pallett
Introduction
Welcome to the first part of a new two-part series on the PHP:Form web component. In this part, I will give you an introduction to PHP:Form, its features, and why it is so useful. I will also discuss the inbuilt validators that PHP:Form supports. In the second part I will discuss the more advanced features of PHP:Form.
What is PHP:Form?
PHP:Form is a (new) web component, developed by TGP PHP Scripts . It is designed to help you create forms with a lot less effort. When you're creating a new PHP script, you will undoubtedly have to create forms to allow input to be entered. There is (almost) no PHP script without forms, and forms usually require solid validation to make sure there are no security leaks. It would be nice if those forms are accessible as well, but this often gets forgotten.
We've all had to create forms again and again. And every time we've had to write those validation functions. I'm sure you remember... if (empty($blah)) { echo 'Invalid'; }, etc. After a while it becomes a really boring and mundane task, and most of the time we don't do it properly either. One of the hardest things to do is to return error messages, indicating what is wrong, and allowing visitors to fix their mistakes, without having to re-do the complete form. I used to simply tell them to hit the back-button, after returning the errors, but this is obviously a not-so-good way of doing it.
Thankfully, that's where PHP:Form steps in. It handles all the boring parts for you, and does it properly as well.
PHP:Form has support for inbuilt validators, which means you only have to use a simple HTML-like syntax to add new validation logic to a form. With these validators also comes automatic client-side validation. All the necessary JavaScript is created for you, and there's nothing you need to do. Another remarkable feature of PHP:Form are so-called "formtypes", which are basically form templates. These allow you define form templates, which can be re-used over and over again.
Let's have a closer look at the validation part of PHP:Form; the validators.
Inbuilt Validators & Client-side Validation
Before we look at the validators, let's first look at the basic PHP:Form syntax. It has a really simple syntax, because it's basic HTML. To create a new form, use the tags, like so:
<php:form name="example">
... form html goes here ...
</php:form>
That's all that is really necessary. But you must also tell the form to display, using PHP, like so:
$_FORMS->display ('example');
Those two things are the only things absolutely necessary to create and display a new form. Of course, nothing will be displayed yet because you haven't created any input fields. To see a simple form in action, have a look at demo 1
Let's move on to validators now. Validators are used to validate form fields, and just like the form tags, they are simple html, for example:
<validator for="[fieldname]">Your error message here</validator>
You can either place validators in between the form tags, or outside the form tags. If you place them outside the form tags you must specify the form name (using the 'form' attribute).
A simple form with a validator would look like this:
<?php
// Include PHP:Form
include ('../phpform.php');
// Begin form:
?>
<php:form name="example">
<validator for="name" required="true"><p style="font-weight:bold;color:red;">Please enter your name</p></validator>
Name: <input type="text" name="name" />
<input type="submit" value="Go!" />
</php:form>
<?php
if ($_FORMS->validate ('example') == true) {
// Show POST'ed values
echo '<pre>';
print_r ($_POST);
echo '</pre>';
} else {
// Display form
$_FORMS->display ("example");
}
?>
View live demo
As you can see in the code we created a validator that has the required attribute set to "true". That means that this validator just checks whether the value of the input field isn't empty.
There are 5 different kinds of validators:
- Required: they are used to make sure an input field isn't empty, like I just demonstrated.
<validator for="field" required="true">Please fill in something</validator>
- Numeric: they are used to make sure an input field only contains numbers, and nothing else.
<validator for="field" numeric="true">Please fill in something</validator>
- Regex: they can be used to specifiy a regular expression that an input field must match.
<validator for="field" regex="/test/i" >Please enter 'test' only.</validator>
- Callback: callback validators can take a callback function that is run on the server-side. That callback function is passed the value of the input field, and the function must return true or false. This is used for really advanced validation (and it's likely you will hardly ever use the callback validator)
<validator for="field" callback="is_email">Not a valid e-mail address.validator>
- Name: name validator can be used to display a message or error only when you want to. They can only be shown when you manually show them using the trigger_error ('form', 'errorname') method.
<validator name="mymsg">This is my custom error!</validator>
Then in PHP:
<?php
$_FORMS->trigger_error ('example', 'mymsg');
?>
When using validators, you will probably want to check if a form validates or not. To do this, use the validate() method, as seen in demo 2:
If ($_FORMS->validate('example') == true) {
echo 'It validates!';
} else {
echo It 'doesn't validate!';
}
Client-Side Validation
PHP:Form also automatically generates client-side validation (JavaScript) when using validators. It natively supports the required, numeric and regex validators, but it doesn't (fully) support the callback validator. This isn't really possible either, because the callback validator points to a function on the server-side. But if you create a JavaScript function with the same name as the callback function, it will work, and it will run the JavaScript function you created. This gives you great power, and means you can even using advanced JavaScript functions and Ajax to validate data.
If you would like to see the client-side validation in action, have a look at demo 2 again, and make sure you have JavaScript enabled. You will probably notice how fast the errors are returned, and that no refresh happens at all. That's the client-side validation.
Conclusion
In this first part of the PHP:Form series I have shown you what PHP:Form is: an extremely neat PHP form component, that is really useful for building web forms. I have been using it myself now for a few months, and I still can't get over how great it is. It has really simplified things, and I can focus on the important stuff. If you're still in doubt, have a look at the PHP:Form product page for more information and demo's .
I have also shown you exactly what validators are, and the different types. Validators are the most important part of PHP:Form, and you will probably use them in every form. You can some really interesting things with them, and when you combine a few validators it's possible to create a extremely secure form.
In the next part I will have a look at "form types", the form templates of PHP:Form. I will also have a look at setting default values, using the set_value() method of PHP:Form.
If you're interested in purchasing PHP:Form, don't forget to use the special PHPit coupon code: phpit
PHP:Form product page
About the Author
Dennis Pallett is a young tech writer, with much experience in ASP, PHP and other web technologies. He enjoys writing, and has written several articles and tutorials. To find more of his work, look at his websites at http://www.phpit.net , http://www.aspit.net and http://www.webdev-articles.com
CGI Security Issues
By Richard Lowe
When you are creating or using CGI routines, you must be careful to keep
good coding techniques, security and just plain common sense in mind.
Sometimes you can do things that cause serious unexpected site effects. In
fact, sometimes you may think you are making your CGI routine secure only
to find out it just doesn't work like you expected.
A good example of a this phenomenon is a simple CGI routine called FormMail.
This was written a number of years ago by a fellow named Matt Wright to
allow data to be entered in a form, then emailed to a recipient.
I first looked at FormMail because I wanted to cut down on spam. You see, my
web site had my email address embedded on every single page. I thought this
was a good idea to allow people to send me an email message when they wanted
to contact me. In fact, all of the web design books indicate that all good
web sites include an email link of this kind.
I soon discovered, much to my horror, that spammers use special programs
called Spam Harvesters to scan websites for email addresses. They add these
addresses to their mailing lists and resell them over and over. The result
is a large increase in the amount of spam that I received.
After much research, I came to the conclusion that the best defense against
spam robots was to simply stop including my email address on my web sites.
This left the question of how to allow users to contact me when they had
questions or comments.
The answer is simple - use a form. The advantage is that the email address
is hidden within the CGI routine or a text file and it is simply not
possible for a spam harvester to pick it up. As long as the email address is
coded into the CGI routine or in a database you are relatively secure.
However, many people use FormMail in a different way. Let's say you want to
allow your visitors to "tell a friend" about your site. So you include a
form which allows visitors to enter their message and a target email
address. If you are not very careful you could find that you have set
yourself up as a spam relay.
You see, spammers are always looking for ways to hide their identity. One
common method is to search the internet for occurrences of FormMail.
Sometimes I wonder if spammers rub their hands together in glee when they
find sites which use FormMail with user-entered email addresses.
The spammer essentially "hijacks" the FormMail CGI routine and causes it to
send out emails as fast and furiously as they can. I know of one instance
where a spammer sent over one million emails in a single day before someone
noticed that their web server was going very slowly (I wonder how long it
would have taken had the spammer tried limiting the load on the server so it
didn't show up as much).
What happens here is very simple. The FormMail CGI routine is simply called
remotely by the spammer, once for each spam email that he wants to send.
Ah, you say, but you could code the FormMail routine to check the referrer
field. This would surely prevent a spammer from using it remotely, as his
referrer would not be the website URL.
Sorry, no. The referrer field is actually a text string passed to the CGI
routine by the browser. The spammer is most likely using a program which
appears, to your web site, to be just another browser. Since the spammer
controls the program he can code it to send the CGI routine whatever value
he wants for the referrer field.
As it turns out, it is very difficult to make a CGI routine such as FormMail
even relatively secure, and it may be impossible to make it bullet-proof.
All you can do is check enough things and put in delays here and there to
slow down and discourage spammers.
You could, for example, only allow one posting per IP address per hour. You
could also check referrer just to block out the more ignorant spammers. I
suppose you could count the number of times the routine is called, and have
it just stop working after a certain amount. For example, only allow one
hundred calls per day from anywhere.
The point here is not to tear apart the FormMail routine. The goal is to
show how difficult it can be to make anything secure on the internet, and
demonstrate that some assumptions (that the referrer field is a valid check)
may not be true in all cases.
What do you do? Before you implement any CGI or similar interface, be sure
and do a little research to be sure you completely understand and handle
the ramifications. If you don't do this, you may find yourself the victim of
a hacker or spammer.
About the Author
Richard Lowe Jr. is the webmaster of Internet Tips And Secrets
at http://www.internet-tips.net - Visit our website any time to
read over 1,000 complete FREE articles about how to improve your
internet profits, enjoyment and knowledge.
|
|
