Web Development Insight

The idea behind robots.txt is to keep honest robots out of certain areas of a web site. One file not to include in robots.txt is…robots.txt. The bots need to read robots.txt to find out their restrictions. A robots.txt is assumed to exist in the root directory of a web site.

Another file to leave out of robots.txt is the sitemap file: generally sitemap.xml and/or sitemap.xml.gz. It is imperative that the bots find this file so they can pare down the amount of indexing needed.

Other files not to exclude from bots are verification files: the ones created to prove you have access to the web site. Some common applications are Google sitemap verification, McAfee Site Advisor, and Yahoo Site Explorer.

Doug

I went to install Expression Web but found out it would not install over the beta version. But the beta version would not uninstall - it was corrupt. Searching the web only found other people with the same dilemma…so I thought I’d try a fix myself.

Note that I did -not- have Office 2007 or Office 2007 beta installed - just Expression Web beta 1 - so I’m not sure how this fix will work if you have Office 2007 or its beta installed.

I had Expression Web Beta 1 but I believe the fix will be similar to the CPT 1 release.

*** Make sure to back up your registry before doing this! ***

1. In the %temp% directory (start > run > open then type in %temp%) search for the latest log SetupExe(..).log. Near the end is this:

Catalyst beta product conflict check failed. Office Beta product conflict is detected on the computer, productcode={30120000-0026-0000-0000-0000000FF1CE}

2. Search for the productcode in the registry and delete all the occurrences but when you find the occurrence in a subkey of:

HKEY_LOCAL_MACHINE/SOFTWARE/Microsoft/Windows/ CurrentVersion/Installer/UserData/
S-1-5-18/Products

you need to delete the entire subkey. In my case their were 4 of them that started with 000021 and contained the productcode in question.

For example:
In …/Products there were 4 keys starting with 000021…. Each key had the productcode mentioned above in the InstallProperties subkey. Delete the entire key starting with 000021… not just the InstallProperties subkey.

There were two places that had these 4 keys to be deleted.

Then delete all occurrences of the productcode in the registry…most were in the Uninstall key.

What this whole exercise does is to delete references to Expression Web beta and Office 2007 beta.

Hopefully these instructions make sense. I did not record every place that I deleted the productcode…I wasn’t sure it would work…but it did.

Doug

Gold has always been a hedge against inflation while silver has been based on industrial use. Gold and gold coin have bought the passage to safety where other forms of money have failed. Since the United States operates in an inflation mode the yellow metal has has an allure of some protection. While most people are against inflation, unfortunately it is ingrained in our life.
Many financial planners have recommended precious metals such as gold and silver as part of an overall investment portfolio. Consider one aspect of portfolio hedge and purchase gold, whether it be American Eagles, Krugerrands, or Maple Leafs for an adjunct to long-term financial security. Ask your financial adviser before investing.

The idea behind the PayPal verification seal is to provide a would-be PayPal customer verification that the PayPal account is in good standing. Click on the verification seal, login to PayPal, and find out the status of the account. The instructions on how to put the seal on a website and how to use it have been obfuscated at PayPal.

The friends at Yahoo group NetBusinessSuccess have figured out how to do this.

1. Download the PayPal seal and put a link to it on your web page.

2. Make the seal clickable to the following URL (change the email address to your PayPal email address)

<a href="http://www.paypal.com/us/verified/pal=abc@mydomain.com"></a>

3. The code should look similar to this:

<a href="http://www.paypal.com/us/verified/pal=abc@mydomain.com">
<img src="http://mydomain.com/images/pp-verification-seal.gif"
 alt="PayPal Verification Seal" width="100" height="100" /></a>

Note at this juncture that PayPal verification is only for United States accounts.

[addition 21 Feb 2007]
If you need to hide your email address from spambots here is a tool to do that:

Email Address Encoder

Doug

One of the banes of blogging is spam comments. Nearly every blogger gets them. One way to reduce comment spam is to turn off accepting comments on older posts. Each blog host will have a different way to do this…if it is possible. Check with your host to see how.

Generally trackbacks are a nuisance since they do not require a login to implement. The poor moderator or comment filter must decide whether these are legitimate or not.

Since shutting off accepting comments, trackbacks, and pingbacks before a specific date is not built-in, WordPress has a comment timeout plug-in that does the job.

Doug

One addition I have found useful is to add email notification when my custom 404 page is invoked. Since I use a PHP file for the custom 404 adding email notification is pretty easy. I renamed the original custom404.htm to custom404.php then added the email code.

The email sent will spell out the invalid page that caused the 404 error. BTW, 404 is the error generated by HTTP when an invalid or nonexistent web page is trying to be accessed. The commented out echo code was used for diagnostics.

Note that nothing is shown on the custom 404 page. The email occurs behind the scenes.

Here is the code:

<?php
// send error email
$em = "someone";
$em .= "@";
$em .= "mydomainname";
$em .= ".com";

$subject="Custom404";
$message="A custom 404 page was accessed from mysecurepc site";
$message .= " The page accessed was: " . $_SERVER['REDIRECT_SCRIPT_URI'];

if(mail("$em","$subject","$message","From: $em")){
//echo "e-mail sent !!!";
}else{
//echo "error !!!";
}
?>

A few things I have noticed since I set this up on a couple of web sites for testing.

1. For some reason, I get a custom 404 email when a bookmark like

...domain.com/folder/webpage.html#topofpage

is accessed. I have not been able figure out why, yet.

2. If you do not have a favicon.ico then the custom 404 page is invoked and you will get email. This occurs when IE accesses the home page.

3. Deleted web pages may show up in an 404 email since search engines retain these links for a while. The searchbots invoke these.

4. Deleted or name-changed web pages may show up because someone is using an old link.

Overall it has given me valuable feedback despite some unwanted emails.

Doug

Gold has always been a hedge against rampant inflation. Gold and gold coins have bought the passage to safety where other forms of money have failed. Since the United States has operated in a pro-inflation mode (rather than deflation) the yellow metal has has an allure of some protection. While most people are against inflation, unfortunately it is part of our life. Many financial planners have recommended precious metals such as gold as part of an overall portfolio. Consider one aspect of portfolio hedge and purchase gold, whether it be American Eagles, Krugerrands, or Maple Leafs for an adjunct to long-term financial security. Ask your financial adviser before investing.

This only works for an Apache server. Note that the folder and its contents will not be indexed since it is passworded.

1. create .htaccess and .htpasswd as ordinary text files. Make sure there is no .txt at the end.

here is an .htaccess example

AuthUserFile /home/content/f/r/e/fred/html/myfolder/.htpasswd
AuthGroupFile /dev/null
AuthName "Access for Admin"
AuthType Basic

<limit GET POST>
require user myfriends
</limit>

where myfriends (replace with your own) is the login id and the folder to protect is myfolder (replace with your own directory path)

line 1 of .htaccess needs to be replaced with the full path on your server to your .htpasswd folder. This is dependent on your ISP. The example shown below is from GoDaddy. The path to the root of your GoDaddy account looks like:

/home/content/f/r/e/fred/html/

This is known as the DOCUMENT_ROOT in the world of PHP. Since most hosts support php you can create a file called showinfo.php and put the following into it:

<? phpinfo(); ?>

Upload this to your website’s home directory and invoke it:

http://www.mydomain.com/showinfo.php

Look for the entry called “DOCUMENT_ROOT”.
for example if DOCUMENT_ROOT was /home/content/f/r/e/fred/html/ and the folder you are trying to protect is myfolder then the entry for AuthUserFile is:

/home/content/f/r/e/fred/html/myfolder/.htaccess

Ask your ISP if you cannot find the full path.

Here is an .htpasswd example

myfriends:bZTGwg.9OWALY

• where myfriends is the login id and the stuff after the colon is the encrypted password.
• make sure .htpasswd is one line only with no breaks at the end

to get the .htpasswd entry you need to generate a password. Try this link:

Generate .htpasswd

It will generate a cut-and-paste line for .htpasswd. Remember no breaks at the end of the line.

If you have access to Perl, try this from the command line:

perl -e "print crypt('myfriends', 'abc')"

where the ‘myfriends’ is the login id and ‘abc’ can be any sequence of letters or numbers. It provides a starting point for the password.

2. upload .htaccess and .htpasswd to the folder you want passworded. If you have a problem uploading try this link:

Uploading the .htaccess file

3. you are done. Every time someone tries to access the folder or any of its subfolders they will be prompted for a login and password.

Doug

Like most people, I have experienced a very frustrating waiting period after I hit Publish in WordPress. It takes forever sometimes. Or the white screen of death: MySQL database access has timed out (or a similar message). Why is this happening?

The default for WordPress is to ping the pingomatic website after a post is published. Look under Options > Write and look under Update Services. There is a reference to pingomatic which pings many, many servers. When all the pinging is done then control is returned…if ever. The default timeout is 15 seconds.

There are several ways around this.

1. Have a service, such as FeedBurner, do your bidding. Delete the entries in the Update Service area and have the service (FeedBurner e.g.) handle your feed. FeedBurner has an option called PingShot which, when activated, will ping pingomatic and other servers. I use a plugin called subscribe me which routes your blog’s feed requests to FeedBurner.

2. A plugin called wp-no-ping-wait will ping the list in Update Service *after* control is returned to WordPress. Thus after Publish button is pressed control is returned almost immediately. The website that handles the plugin has been down for a while so go to this link to get it (the wayback machine). Look for nopingwait2.php. Upload it to your plugin directory and activate it.
3. Delete all entries under Update Service and manually ping the servers yourself using pingoat or pingomatic.

Doug

Most online people have their own web sites. For business owners web site is almost mandatory. One of the best ways to improve link popularity and traffic of a website is by blogging. Up to date information can be posted quickly, giving viewers an edge. A way to increase quality links to your web site is to have others write blog articles about your niche. This service is a way to quickly get many links pointing to your web site. Search engines use quality links as an important part of their algorithm for rating a web site. This is one avenue worth investigating.

Doug