Re: using sockets to open connection to a search engine
Damo wrote:
Hi,
I'm trying to open a connection to altavista.com through java to
retrieve the search results for a query. This is the code I'm using, it
works for google and yahoo but not altavista or MSN.
s = new Socket("altavista.com",80);
p = new PrintStream(s.getOutputStream());
p.print("GET /web/results?q=java HTTP/1.0\r\n");
p.print("User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.1) Gecko/20061010 Firefox/2.\r\n");
p.print("Connection: close\r\n\r\n");
in = s.getInputStream();
If you type this : www.altavista.com/web/results?q=java into
the address bar, it will return the result page.
Put something in between your browser and AltaVista and
see what the browser sends.
You already have User-Agent, but maybe it wants Referrer or
Accept or Accept-Language or Accept-Encoding.
Or maybe it wants HTTP/1.1 (which requires Host).
There is a limited number of things to add until
you are fully browser compatible.
Arne