Posted by jonvon on May 18, 2002 at 14:27:32:
In Reply to: Enter! Childish Frumpkin...... posted by giveawayboy on May 04, 2002 at 21:29:09:
bill, man i love your poem, and i love this chat you pasted here. VERY COOL.
this is something i have thought a lot about. i haven't worked on any actual engineering of any of these things, but being someone who does "web stuff" for a living, i've followed the evolution of search engines indirectly for a long time.
this whole thing about inserting people in between machines and people who are searching for data in the machines, this is interesting. man, there is a whole lot of history about that. actually the most successful search engine ever (until very recently) was Yahoo!, and Yahoo isn't really a search engine, its a directory put together by people. There is a directory now called the Open Directory Project where submissions for new web sites are taking place, and people are volunteering all the time to kind of be the librarians that John mentions downthread.
...you could volunteer if you wanted to get involved.
of course the big gun these days is Google. and they have inserted the people into the equation in a very simple and interesting way. part of the algorithm they have devised to rank search results is links. basically, the more people are linking to a site, generally speaking, the higher the ranked result will be. there is way more to their intelligence than that, but that is the basic premise. when you go to search for something, usually the subject you are searching for has been written about, and someone else who is probably more interested in it than you are has made a link or a list of links to what they think are good sources of information, and then google finds those links, and wa la... so the people who are the most expert about something end up inadvertantly creating the basis for the relevancy rankings of any particular set of search results.
there isn't any way a group of librarians (like the people putting the directory at Yahoo together by hand) could ever compete with that kind of power. the internet just moves too fast, people are adding more and more data all the time. the directories aren't going away, in fact if you want to submit your site for the Google spider you have to go through Open Directory to do it. but about the best a directory, even one with lots of money like Yahoo, can do is to catalog a very small percentage of the internet. and the bigger it gets, the smaller that percentage will be.
people have been flocking to google because as it turns out the search results are very very good, really better than anything else available.
Google is so good, often if i am searching for something, even using a site search that is specific to a particular site, Google will give me better results. This happened the other day on that Wired site. i was looking for an article, couldn't find it with the tools Wired provided, I went to Google, and a few seconds later i found the article on Wired's site.
another interesting search engine that does very well with relevant results is AllTheWeb. it uses artificial intelligence to configure relevancy, and it does a pretty good job. i go there once in a while, but usually Google is just too good and too fast (they have a gigantic network with thousands of servers), i don't need anything else.
google also employs another "behind the scenes" technology called Pigeon Rank. Very interesting stuff.
ok, but seriously, something else i just ran into on the ask.com site is something they call Answer Point. You can sign up and then for really tough questions that you can't seem to find the answer for you can ask in a forum setting. Then hopefully someone will answer your question.
The is another thing you can get involved with if you are interested, and become a person who answers questions. actually you can get to this directly at http://answerpoint.com
To me, these kinds of ad hoc communities are the most powerful. Somehow that old axiom that everyone on the net like to quote, "information wants to be free", that is true. The more search engines and the tools they employ reflect the community as it has built itself, as it behaves, and as it morphs, the better the engine will be.
I'm looking around for a site i used to surf once in a while. it was really neat. people would enter terms for all kinds of things, might be pop culture terms/entities/whatever, might be stuff that only exists on the internet, might be practically anything as long as it was valid. i think there was a review process of some kind. i wish i could find it, or remember what it was called. ironicly, the mighty Google hasn't turned up anything for me. i'm sure i'm not asking the right questions...
anyway, all that to say that really since the beginnings of the net people have been thinking about the "interface between fields of knowledge and people". it is the number 2 question (in my mind anyway - for people who are involved in creating the net) behind, how do i get this information i have out there. it might be number 1 really, with the other one as number 2.
basically the net isn't worth much without these tools. if its out there but you can't find it, what good does it do you?
thanks for posting on this, very cool.