Intelligent Ways Of Improving Page Rankings
Experts feel that web pages have to be search engine friendly in order to improve search engine rankings. Having a well designed and relevant web copy is crucial. This will help search engines easily index your web pages and rank them high.
Besides internal page factors including frequency and positioning of relevant keyword phrases, the leading search engines now suggest the importance of off the page factors, particularly links from other domains.
Google ranks individual URLs based on which other URLs link to them, which URLs link to those, and so on. Thus the steps you have to take are:
Choose the terms that you wish to include as the most productive search terms for your site, terms that you associate with the correct target visitors to your site. Put those in the search engines and find what pages rank near the top for them. These top-ranking sites/pages would be on the top of your list, if you needed to plant links to your site on other pages.
Once you have found the top ranked pages/sites for your chosen search term(s), you should check what sites or pages link to that site. If you get links from any of these sites to your target pages, you may be able to dramatically improve your ranking on Google and other crawler based Search Engines.
If a website has a lot of interlinked pages, then even an obscure page from that site could be a reasonably effective candidate for having a link to your site. It is not essential that the link should be on the home page or prominent page of a popular site.
Creating fake domains and letting those domains point to your site is a trick thought to be effective for improving page ranking at Google. However Google claims that this is not true and they are able to spot duplicate domains and domain scams.
1.1 Link Popularity
Your "link popularity" is simply a count of the number of web pages that are linked to you. Improving your website’s link popularity is absolutely vital for improving the visibility of your website with regard to Search Engines. You may want to know your link popularity for two reasons. The first is that your link popularity will improve your ranking on all crawler search engines. As discussed earlier, all crawler-based search engines have a component called the spider, which crawls from one webpage to another through links. Hence, more the websites linking to your website, the better are your chances of getting listed through a search engine. The second reason is that you might want to know which websites are linked to you and potentially referring traffic.
Search engines give sites with good inbound and outbound links a higher ranking. The logic goes that if you provide outbound links to other material you are providing a valuable service: and, if other sites link to you then you must have content of value.
1.1.1 Link Analysis and measuring Link Popularity
The best way to discover how people are finding your web site is to analyze your site's activity logs. If you are unable to analyze their logs can instead use search engines to track down referral links. In particular, this method gives you an idea of how "popular" a search engine believes your site to be. Be aware that "popularity" is only one part of the link analysis systems that search engines such as Google use to rank web pages. The quality and context of links is also taken into account, rather than sheer numbers.
You can use link:’site URL’ feature of many search engines to list all the pages that link to the selected site, and that too in order of Page Rank. For Google, North Light and AltaVista, use link:xxyyzz.com to find the listing of pages that link to the web site www.xxyyzz.com For Alltheweb use link.all instead of link and for Inktomi use
‘linkdomain’ instead of ‘link’ in the above example. The results would be a list of all pages (if indexed by the search engine) that link to your target site, listed in the order of popularity.
If you need to find the link to specific pages instead of to an entire site, then the above link: feature will not work. Use the Advanced search features offered by HotBot and MSN Search, enter the full URL of the target page including http:// and use the option “links to URL” or similar.
Some sites offer to run comparison of the links to a chosen site vis-à-vis three other chosen sites. http://linkpop.marketleap.com and http://www.linkpopularitycheck.com are two such sites where you could submit your target URL and three other URLs that you wish to have a comparison done. www.linkpopularity.com is a site that will analyze the link popularity of a chosen URL in three prominent search engines.
Link analysis is somewhat different than measuring link popularity. While link popularity is generally used to measure the number of pages that link to a particular site, link analysis will go beyond this and analyze the popularity of the pages that link to your pages. In a way link analysis is a chain analysis system that accords weighting to every page that links to the target site, with weights determined by the popularity of those pages. Search engines use link analysis in their page-ranking algorithm. Search engines also try to determine the context of those links, in other words, how closely those links relate to the search string. For example if the search string was “toys”, and if there were links from other sites that either had the word toys within the link or in close proximity of the link, the ranking algorithm determines that this a higher priority link and ranks the page, that this is linked to, higher.
As a site owner, you want to seek links from good pages that are related to the terms you want to be found for. Linking strategy is not a trick as many get rich quick merchants would have you believe. Links for the sake of links have no value whatsoever. Indeed, they can damage your rankings. So forget about link farms and other such nonsense. A small number of inbound links from great, relevant sites will be much more valuable than many links from low-traffic, irrelevant sites.
However, you should not become obsessed by link popularity alone. Treat linking as one important aspect of your Search Engine Optimization strategy. Decide how much time and effort you are prepared to invest in relation to your other activities and be disciplined about your approach. Monitor your results and adapt your strategy as necessary.
1.1.2 Where and how to seek reciprocal linking
Once you have found the candidate sites that have high link popularity and link quality as seen through link analysis, the next step is to choose those that you believe may agree to reciprocal linking. Your competitors obviously would not, but in respect of others, you must try. Several sites have a page where they list useful links or relevant links, as a service to their site visitors. You can locate the email of the company or of the person who handles link requests for such list pages.
When you make a request for reciprocal linking, approach the target site owner or webmaster or link request handler by providing your URL and a short description and explain how providing this link would be a valuable and useful addition for the benefit of their audience. The description is important as often that would be what appears in their links page. Offer them reciprocal linking from your site’s links and resources page. Better still tell them that you have already provided a reciprocal link to their site.
Build a good links and resources page on your website. Present the links in an organized manner so that this is useful to your site visitors too. Of course, you do not want this links page to be amongst the first pages accessed by your visitor, as this may induce the person to leave your site. Some webmasters try to build a standalone links page that is totally isolated from all other pages on their site. This would perhaps provide an URL to the reciprocal link provider, but in reality it is not a genuine link page at all. Avoid these tactics; they will not work in the long run.
Finally, not all sites are equal and therefore not all links are equal. A link from a high traffic industry portal is worth infinitely more that a link from a low-traffic free-for-all site. Concentrate of giving the search engines what they really want - great content, well- organized, well-published and linked to other relevant material. Concentrate on that and you will be rewarded.
1.1.3 Inward Linking
Like reciprocal linking, inward links to your website can be an effective strategy to increase your website’s visibility to Search Engines. Inward links are links pointing to your websites from other websites without providing a reciprocal link from your website.
There are many techniques to improve inward linking. Many of these have enjoyed success with Search Engines. The most proven technique for inward linking is through Ebooks. You can offer interesting and educative Ebooks for free to other websites and they could install them on their sites. The Ebook you create would have a link to your website. This will allow a spider to crawl through that link and visit you website. For example, a footer on every alternate page can have a link to your website that would increase the probability of your website being listed with a crawler based Search Engine.
Other techniques include posting newsletters, white papers, news stories and press releases to other websites, particularly industry specific and general portals. The newsletters and press releases would contain a link pointing to your website, thus, increase its visibility to crawlers.
Affiliate programs also help in improving inward linking. In affiliate schemes, you provide incentives (usually a commission on the sale of your product or service) for other websites to become affiliates (i.e. carriers). These affiliates then generate you direct traffic; the added bonus comes in the form of those inward links to your site. Affiliate Programs create powerful alliances between your web site and your various "affiliate" web sites. Providing affiliate links to your website would improve your website’s search engine ranking. It makes your website more visible to crawlers.
1.2 Meta tags and keywords
While Meta tags are not the complete answer to the question of "How do I improve my search engine ranking?” they can help with some search engines. Since there are millions of pages with Meta tags, you can add all the pages you want and still not control a sizeable percentage of the pages on the World Wide Web.
What are Meta tags? They are information inserted into the "head" area of your web pages. Other than the title tag, information in the head area of your web pages is not seen by those viewing your pages in browsers. Instead, Meta information in this area is used to communicate information that a human visitor may not be concerned with. Meta tags, for example, can tell a browser what "character set" to use or whether a web page has self- rated itself in terms of adult content.
Meta tags may help you with some search engines, so you’ll want to consider adding them to every page you create. On the other hand, you can find many highly ranked web pages without Meta tags. For example, Meta tags have no effect on how humans will view your pages and enter your information into directories like Yahoo.
1.2.1 Tags that do matter
Titles
The TITLE tag is an important one for search engines like Google, as that is often the first one indexed and it is given higher weighting in the relevance rankings. You have to pay attention to this tag. Keep it short (lower than 40 characters) and let the tag have material relevant to the keywords used during search. As an example, if you had stuffed your Company Name as the TITLE tag, it would not help. The visitor that you want to attract is unlikely to be looking for your Company name in the search string. Think creatively as to what keywords people would use if they were looking for goods or services that your site offers.
Keywords and Description
Two Meta tags that are important are KEYWORDS and DESCRIPTION. You have to be very careful about how these are developed and positioned. Frequency and location of the keyword that is being searched are important criteria that determine relevance and hence page ranking. The search engine would generally consider the page more relevant if the keyword that is being sought is in the TITLE tag or is in the KEYWORDS tag near the top of the page. Similarly if the sought after keyword is found being repeated in the page, it may give the impression that this is a more relevant page and improve ranking.
There is a caveat, though. The above is only a general rule that has been often followed by many search engines, but then there are many variants to it. Several players in the SEO industry have tried to proclaim that this is the gospel truth and hence it has spawned a large number of experts that suggest and resort to keyword stuffing and spamming (repeating long strings of keywords). The result can often be just the opposite. Some search engines penalize pages that have keyword spamming. Some will just ignore these pages. Some engines also do not read Meta tags. The intelligent method today is to stay away from spamming and to use tags judiciously. Blend your technique to attain the right frequency and location, but stay away from any excess or spamming.
You can provide an in-depth list of words and phrases in the KEYWORD Meta tag. These words should have some relevance to the specific page or, at least, to your website. While you can vary the case of the keywords, you’ll want to concentrate on the lower case because over 90% of the searches either use lower case or are conducted on search engines that are not case sensitive.
These keywords should contain variations on the same theme. If your site was about gardening, you could use garden, gardening, home and garden, home gardening, vegetable garden, and herb garden. These are all words that might be used in searches for information that your site might provide. The keywords Meta tag is not intended to replace the actual text on your website. This tag is simply to aid the spider in collecting accurate information about your web pages.
The DESCRIPTION tag is used by search engines like Inktomi for the page summary that is displayed on the results page. This summary is what the visitor will read and decide whether he/she wishes to enter your site. If the description is just full of repeated
keywords, it won’t do you any good, even if your page is ranked high. You still do not have a visitor. May be you have put off a visitor.
Meta Robots Tag
This is probably the only other prominent Meta tag. This is a very peculiar tag in the sense that it indicates what web pages should not be indexed by Search Engines. The Robots tag is inserted in between the header tags. An example of the Robots Tag is given below:
<HEAD>
<TITLE>This page should be kept out of Search Engine listings</TITLE>
<META NAME=”ROBOTS” CONTENT=”NOINDEX”>
</HEAD>
By default, a crawler will try to index all your web pages and will try to follow links from one page to another. This can be prevented by using the Robots Tag. Most major search engines support the Meta robots tag. The Meta robots tag also has some extensions offered by particular search engines to prevent indexing of multimedia content.
There are other Meta tags apart from the ones explored above but most of them are simply ignored by almost all search engines.
To sum up, some search engines will give you a boost if you have Meta tags. But don't expect that to necessarily be enough to put you in the top ten. Meta tags are mainly a design element you can tap into, a crutch for helping information-poor pages better be acknowledged by the search engines.
1.2.2 Choosing keywords that are focused and specific
This is an important step in the SEO exercise. What keywords do you need to emphasize and include? How do you choose the most relevant keywords that will be used by your target audience? Pose yourself the question: What would my target visitor be looking for and for which queries I would like to lure him/her to my site? Imagine those queries, as many of them as you can think. Apply this to all categories of visitors that you are targeting. Then list those queries and formulate your keywords from those.
Let your creativity develop grammar and synonym variants of these keywords. The same would apply to descriptions; however while keywords have to match closely with search strings, descriptions should be drafted to allure the visitor after he has seen your listing. It should tell the visitor that your site is indeed offering him best information or outcome for what he is looking.
Select phrases (at least two words), rather than single words, as there would be too many contenders for single word searches. You are much better off focusing on specific search keywords and using longer phrases.
1.2.3 Location of keywords
The location of keywords on your website is vital. You may have placed keywords at only a few locations but if they are inserted at important positions, your search engine ranking may be boosted. The most important position is the Title of the page. Ensure that your keyword is placed in the Title.
Another proven strategy is to place keywords towards the beginning of the web page. For instance, inserting keywords in page headlines, other sub-titles, and introductory paragraphs would certainly help rankings. Avoid using images or tables at the beginning of the page. This will only shove keywords down on the web page, and in turn harm your search engine rankings.
Amongst other strategies, “stemming” is another important technique for placing keywords. This is discussed in detail later in this chapter.
1.2.4 Making keywords relevant and consistent
Positioning Meta tags and keywords into web pages for better ranking should not be confused with spamming. Whatever you do, avoid spamming and keyword stuffing. Search Engines have mechanisms that can spot spamming and as a result ignore such web pages. The key is to insert just “enough” keywords into the web page so as to make the content relevant. Keywords have to be consistent with the entire content of the web page. In other words, do not insert non – relevant keywords just for the sake of improving page rankings.
Many webmasters believe that adding more graphics to a web page would make it more attractive. This is a misconception. A web page should be rich with HTML text rather than graphics. This makes the site more relevant as Search Engines can easily scan text.
1.2.5 Stemming and other word forms
Search engines often look for variants of words from a stem and this is referred to as stemming. Thus ‘play’ can also lead to a search for ‘plays’ or ‘playing’ or ‘players’. Similarly singular and plural forms of words and case can lead to different results for some search engines. Some advisers suggest that Capitalization is a better bet in choosing keywords. However, the wiser counsel is that these variants do not make that great a difference and one should prefer lowercase. Most searches are made in lowercase. Moreover, many search engines are not case-sensitive.
1.2.6 Focus on important keywords: Finding top keywords
You may like to know what are the most popular search words and strings, so that you could choose top keywords related to your own site. Some resources that help in this are discussed here.
Place your subject term in Overture’s free Search Term Suggestion Tool. The result will be a list of all search terms related to the word you inserted that were most popular at Overture. The list is in the order of popularity. Thus you know what most people are looking for. Wordtracker has a fee-based service that lets you do the same based on Meta search engines such as Dogpile and Meta Crawler. Supplement this effort through the Related Searches feature that is seen in many search engines. Sites such as AltaVista, Yahoo, HotBot and others have this feature. Place your subject or one of the target keywords in the search string on these. From the results page go to the section: “Others searched for” or “Related searches” that is available on many search engines and you will find the other related terms listed there.
1.3 Write a persuasive home page
In the field of search engine optimization (SEO), writing a strong homepage that will rank high in the engines and will read well with your site visitors can sometimes present a challenge, even to some seasoned SEO professionals. Once you have clearly identified your exact keywords and key phrases, the exact location on your homepage where you will place those carefully researched keywords will have a drastic impact in the end results of your homepage optimization.
One thing we keep most people say is that they don’t want to change the looks or more especially the wording on their homepage. Understandably, some of them went to great lengths and invested either a lot of time and/or money to make it the best it can be. Being the best it can be for your site visitors is one thing. But is it the best it can be for the search engines, in terms of how your site will rank?
If you need powerful rankings in the major search engines and at the same time you want to successfully convert your visitors and prospects into real buyers, it's important to effectively write your homepage the proper way the first time! You should always remember that a powerfully optimized homepage pleases both the search engines and your prospects. In randomly inserting keywords and key phrases into your old homepage, you might run the risk of getting good rankings, but at the same time it might jeopardize your marketing flow. That is a mistake nobody would ever want to do with their homepage.
Even today, there are still some people that will say you can edit your homepage for key phrases, without re-writing the whole page. There are important reasons why that strategy might not work.
Your homepage is the most important page on your web site
If you concentrate your most important keywords and key phrases in your homepage many times, the search engines will surely notice and index it accordingly. But will it still read easily and will the sentences flow freely to your real human visitors? There are some good chances that it might not. As a primer, having just 40 or 50 words on your homepage will not deliver the message effectively. To be powerful and effective, a homepage needs at least 300 to 400 words for maximum search engine throughput and effectiveness.
One way to do that is to increase your word count with more value-added content. This often means rewriting your whole homepage all over again. The main reason to this is you will probably never have enough room to skillfully work your important keywords and key phrases into the body text of your homepage. This may not please your boss or marketing department, but a full re-write is often necessary and highly advisable to achieve high rankings in the engines, while at the same time having a homepage that will please your site visitors and convert a good proportion of them into real buyers.
Here is the acid test that will prove what we just said is right: Carefully examine the body text of your existing homepage. Then, attempt to insert three to five different keywords and key phrases three to four times each, somewhere within the actual body of your existing page. In doing that, chances are you will end up with a homepage that is next to impossible to understand and read.
One mistake some people do is to force their prospects to wade through endless key phrase lists or paragraphs, in an attempt to describe their features and benefits. The other reason they do that is in trying to please the search engines at the same time. Writing a powerful and effective homepage around carefully defined keywords and key phrases is a sure way you can drive targeted traffic to your web site and keep them there once you do.
If some people still say re-writing a homepage takes too much time and costs too much money, think of the cost of losing prospective clients and the real cost of lost sales and lost opportunities. In the end, writing a strong homepage that will achieve all your desired goals will largely justify your time invested and the efforts you will have placed in the re- writing of your homepage.
1.3.1 Setting a theme for your Homepage
We discussed the importance of the Homepage. This section presents a recommended layout for your homepage in order to make it as search engine friendly as possible. This is where you set the theme of your site. Let's suppose the primary focus of your site is about online education. You also have secondary content that is there as alternative content for those not interested online education. There is also other content that you would like to share with your visitors. For example, this might include book reviews, humor, and links.
The top of your homepage, as discussed earlier is the most important. This is where you set the keywords and theme for the most important part of your site, the thing you really want to be found for. So you might start off the top of your index page something like this:
(After your logo or header graphic)
1) A heading tag that includes a keyword(s) or keyword phrases. A heading tag is bigger and bolder text than normal body text, so a search engine places more importance on it because you emphasize it. Heading sizes range from h1 - h6 with h1 being the largest text. If you learn to use just a little Cascading Style Sheet code you can control the size of your headings. You could set an h1 sized heading to be only slightly larger than your normal text if you choose, and the search engine will still see it as an important heading.
2) Next would be an introduction that describes your main theme. This would include several of your top keywords and keyword phrases. Repeat your top 1 or 2 keywords several times, include other keyword search terms too, but make it read in sentences that makes sense to your visitors. A second paragraph could be added that got more specific using other words related to online education.
3) Next you could put smaller heading. Then you'd list the links to your pages, and ideally have a brief decision of each link using keywords and keyword phrases in the text. You also want to have several pages of quality content to link to. Repeat that procedure for all your links that relate to your theme.
4) Next you might include a closing, keyword laden paragraph. More is not necessarily better when it comes to keywords, at least after a certain point. Writing "online education" fifty times across your page would probably result in you being caught for trying to cheat. Ideally, somewhere from 3% - 20% of your page text would be keywords. The percentage changes often and is different at each search engine. The 3-20 rule is a general guideline, and you can go higher if it makes sense and isn't redundant.
5) Finally, you can list your secondary content of book reviews, humor, and links. Skip the descriptions if they aren't necessary, or they may water down your theme too much. If you must include descriptions for these non-theme related links, keep them short and sweet. You also might include all the other site sections as simply a link to another index that lists them all. You could call it Entertainment, Miscellaneous, or whatever. These can be sub-indexes that can be optimized toward their own theme, which is the ideal way to go.
Now you've set the all important top of your page up with a strong theme. So far so good, but this isn't the only way you can create a strong theme so don't be compelled into following this exact formula. This was just an example to show you one way to set up a strong site theme. Use your imagination, you many come up with an even better way.
It's important to note that you shouldn't try to optimize your home page for more than one theme. They just end up weakening each other's strength when you do that. By using simple links to your alternative content, a link to your humor page can get folks where they want to go, and then you can write your humor page as a secondary index optimized toward a humor theme. In the end, each page should be optimized for search engines for the main topic of that page or site section.
Search engine optimization is made up of many simple techniques that work together to create a comprehensive overall strategy. This combination of techniques is greater as a whole than the sum of the parts. While you can skip any small technique that is a part of the overall strategy, it will subtract from the edge you'd gain by employing all the tactics
1.4 Using Clickthroughs and Affiliate Programs for better ranking
In affiliate programs, sites that send you traffic and visitors, have to be paid on the basis of per click or other parameters (such as number of pages visited on your site, duration spent, transactions etc). Most common contractual understanding revolves around payment per click or clickthroughs. Affiliates use tracking software that monitors such clicks using a redirection measurement system. The validity of affiliate programs in boosting your link analysis is doubtful. Nevertheless, it is felt that it does not actually do any harm. It does provide you visitors, and that is important. In the case of some search engines re-directs may even count in favor of your link analysis. Use affiliate programs, but this is not a major strategy for optimization.
1.5 Doorblocks
Several pages in e-commerce and other functional sites are generated dynamically and have “?” or “&” sign in their dynamic URLs. These signs separate the CGI variables. While Google will crawl these pages, many other engines will not. One inconvenient solution is to develop static equivalent of the dynamic pages and have them on your site. Another way to avoid such dynamic URLs is to rewrite these URLs using a syntax that is accepted by the crawler and also understood as equivalent to the dynamic URL by the application server. The Amazon site shows dynamic URLs in such syntax. If you are using Apache web server, you can use Apache rewrite rules to enable this conversion.
1.6 Submit a “site map” page or a page with links to all inner pages
One good tip is that you should prepare a crawler page (or pages) and submit this to the search engines. This page should have no text or content except for links to all the important pages that you wished to be crawled. When the spider reaches this page it would crawl to all the links and would suck all the desired pages into its index. You can also break up the main crawler page into several smaller pages if the size becomes too large. The crawler shall not reject smaller pages, whereas larger pages may get bypassed if the crawler finds them too slow to be spidered.
You do not have to be concerned that the result may throw up this “site-map” page and would disappoint the visitor. This will not happen, as the “site-map” has no searchable content and will not get included in the results, rather all other pages would. We found the site wired.com had published hierarchical sets of crawler pages. The first crawler page lists all the category headlines, these links lead to a set of links with all story headlines, which in turn lead to the news stories.
1.6.1 Size of submitted page
We have written above that the spiders may bypass long and “difficult” pages. They would have their own time-out characteristics or other controls that help them come unstuck from such pages. So you do not want to have such a page become your “gateway” page. One tip is to keep the page size below 100 kb.
1.7 Submit only key pages
You do not have to submit all the pages of your site. As stated earlier, many sites have restrictions on the number of pages you submit. A key page or a page that has links to many inner pages is ideal, but you must submit some inner pages. This insures that even if the first page is missed, the crawler does get to access other pages and all the important pages through them. Submit your key 3 to 4 pages at least. Choose the ones that have the most relevant content and keywords to suit your target search string and verify that they link to other pages properly.
1.8 Working with Frames
Many websites make use of frames on their web pages. In some cases, more than two frames would be used on a single web page. The reason why most websites use frames is because each frame’s content has a different source. A master page known as “Frameset” controls the process of clubbing content from different sources into a single web page. Such frames make it easier for webmasters to club multiple sources into a single web page. This, however, has a huge disadvantage when it comes to Search Engines.
Some of the older Search Engines do not have the capability to read content from frames. These only crawl through the frameset instead of all the web pages. Consequently web pages with multiple frames are ignored by the spider. There are certain tags known as
“NOFRAMES” (Information ignored by frames capable browser) that can be inserted in the HTML of these web pages. Spiders are able to read information within the
NOFRAMES tags. Thus, Search Engines only see the Frameset. Moreover, there cannot be any links to other web pages in the NOFRAMES blocks. That means the search engines won't crawl past the frameset, thus ignoring all the content rich web pages that are controlled by the frameset.
Hence, it is always advisable to have web pages without frames as these could easily make your website invisible to Search Engines.
Making frames visible to Search Engines We discussed earlier the prominence of frames based websites. Many amateur web designers do not understand the drastic effects frames can have on search engine visibility. Such ignorance is augmented by the fact that some Search Engines such as AltaVista are actually frames capable. AltaVista spiders can crawl through frames and index all web pages of a website. However, this is only true for a few Search Engines.
The best solution as stated above is to avoid frames all together. If you still decide to use frames another remedy to this problem is using JavaScripts. JavaScripts can be added anywhere and are visible to Search Engines. These would enable spiders to crawl to other web pages, even if they do not recognize frames.
With a little trial and error, you can make your frame sites accessible to both types of search engines.
1.9 Using Robots.txt to your advantage
We discussed the ROBOTS tag in brief earlier. Let us understand this tag a little more in detail.
Sometimes we rank well on one engine for a particular keyphrase and assume that all search engines will like our pages, and hence we will rank well for that keyphrase on a number of engines. Unfortunately this is rarely the case. All the major search engines differ somewhat, so what's get you ranked high on one engine may actually help to lower your ranking on another engine.
It is for this reason that some people like to optimize pages for each particular search engine. Usually these pages would only be slightly different but this slight difference could make all the difference when it comes to ranking high.
However because search engine spiders crawl through sites indexing every page it can find, it might come across your search engine specific optimizes pages and because they are very similar, the spider may think you are spamming it and will do one of two things, ban your site altogether or severely punish you in the form of lower rankings.
The solution is this case is to stop specific Search Engine spiders from indexing some of your web pages. This is done using a robots.txt file which resides on your webspace.
A Robots.txt file is a vital part of any webmasters battle against getting banned or punished by the search engines if he or she designs different pages for different search engine's.
The robots.txt file is just a simple text file as the file extension suggests. It's created using a simple text editor like notepad or WordPad, complicated word processors such as Microsoft Word will only corrupt the file.
You can insert certain code in this text file to make it work. This is how it can be done.
User-Agent: (Spider Name)
Disallow: (File Name)
The User-Agent is the name of the search engines spider and Disallow is the name of the file that you don't want that spider to index.
You have to start a new batch of code for each engine, but if you want to list multiply disallow files you can one under another. For example – User-Agent: Slurp (Inktomi's spider)
Disallow: xyz-gg.html
Disallow: xyz-al.html
Disallow: xxyyzz-gg.html
Disallow: xxyyzz-al.html
The above code disallows Inktomi to spider two pages optimized for Google (gg) and two pages optimized for AltaVista (al). If Inktomi were allowed to spider these pages as well as the pages specifically made for Inktomi, you may run the risk of being banned or penalized. Hence, it's always a good idea to use a robots.txt file.
The robots.txt file resides on your webspace, but where on your webspace? The root directory! If you upload your file to sub-directories it will not work. If you wanted to disallow all engines from indexing a file, you simply use the * character where the engines name would usually be. However beware that the * character won't work on the Disallow line.
Here are the names of a few of the big engines:
Excite - ArchitextSpider
AltaVista - Scooter
Lycos - Lycos_Spider_(T-Rex)
Google - Googlebot
Alltheweb - FAST-WebCrawler
Be sure to check over the file before uploading it, as you may have made a simple mistake, which could mean your pages are indexed by engines you don't want to index them, or even worse none of your pages might be indexed.
Another advantage of the Robots.txt file is that by examining it, you can get information on what spiders, or agents have accessed your web pages. This will give you a list of all the host names as well as agent names of the spiders. Moreover, information of very small search engines also gets recorded in the text file. Thus, you know what Search Engines are likely to list your website.
|
