I love it.

I love having to program something I have never done and having to figure out how the hell to do it. Man it's nice to be able to work symbolically with arguments I put into things like:

document.getElement.ByID(div+i).innerHTML=something.responseText;

and I can fill in a bunch of divs at one time.


Now I will go listen to MiniPop and have their beautiful melodies haunt me.

AJAX

Since I have a lot of time on my hands at this time, I'm getting back into studying code. Like multiple hours daily if possible. Right now I'm implementing AJAX to make a page that deals with AI concepts. Half the page is cellular automata, the other half is evolutionary algorithms. The idea is to calculate and retrieve data behind the scenes. I found a way to do it, but I'd rather come up with my own way, which does not employ ScriptManager.

Favicon is Evil

I am working on the spider again tonight. I learned a few things. Like HTTPWebRequest.timeout exists. And that favicon.ico is a pain in the butt, it always comes up and stops the spider. I've been using Regex to parse strings that have chars I just don't want to be in URLs.

I was reading about other people making web spiders and they are just out grabbing links, while I am recursively searching links to other links. As such, I am verifying that these pages get responses. I don't travel to them if I can't. Although I use a page that provides random links as a starting point, I'm following them as far as I can go.

some stuff:

wreq.Timeout = 60000;
Regex r = new Regex("favicon");
if(r.Success)
etc...

Regexing

I was doing some Regexing last night in dealing with the input string search string. An interesting function it is, that Regex. Basically I want to keep weird chars out.

Not back to the spider yet

What I'm working on tonight is how to rank my results while finding them in the db. Basically, the number of terms found in a result gives a higher rank. Then I plan to sort an array depending on rank for display purposes.

I was just reflecting the other day that there is an AI component to my search machine.

Objects

It turns out you can't go:

row[i+1]++

because objects can't be incremented.
You can get around this by having a temporary variable.

Begin

For the past couple months I have been involved in a big project I set for myself: design a spider and search engine using ASP and C#.Net. I have a functioning spider and currently a search engine with a few things that need to be added. Each works in a limited domain currently. The spider follows links on a page to a next page, grabs the page title and some text, then follows the link to the next page. This occurs recursively, as a page that will not open causes the spider to backtrack until it finds another URL to follow. It can be viewed as a type of tree traversal.

The URLs are stored in a SQl database which will be read from by the search engine. The search engine has its own db for storing info about terms and other secret stuff that makes it work.