Re: HTTPUrlConnection does not download the whole page
On 3 Feb., 17:25, Lew <no...@lewscanon.com> wrote:
The87Boy wrote:
I have a problem with this code, as you can see in print, where it
prints the error in the page's code:
What error? Why not copy and paste the error message in your post so t=
hat we
can actually have a prayer of helping you?
Actually there are no errors, but it fails when I try to print the
page as the System.err.println is printed inside the
System.out.println
public void print(String link) {
String page = this.getPage(link);
You don't need to, and shouldn't, prefix member method calls with "this."=
..
For one thing, it's misleading in the presence of overridden methods, or =
if
'this' class doesn't override the method.
I did not know that, but it is easier to do in Netbeans ;)
try {
URL url = new URL(link);
// Open the Connection
HttpURLConnection conn = (HttpURLConnection)
url.openConnection();
// Set the information
conn.setRequestProperty("user_agent", =
"Mozilla/5.0
(Windows; U; Windows NT 6.0; da-DK; rv:1.9.1.4) Gecko/20091016 Firefox/
3.5.4 (.NET CLR 3.5.30729)");
conn.setRequestProperty("max_redirects", =
"0");
conn.setRequestProperty("timeout", =
"300");
conn.setRequestMethod("GET");
conn.setDoOutput(true);
// Connect
conn.connect();
// Get the Status-Code and add it to the HashMa=
p
int statusCode = conn.getResponseCode();
String page = this.getPage(conn.getInputStrea=
m());
pageEscaped = StringEscapeUtils.unescapeHtml(=
page);
conn.disconnect();
} catch (IOException e) {System.err.println(e.getCause
());System.err.println(e.getMessage());}
You problem stems at least in part that you continue blithely along prete=
nding
to process the URL after you catch an exception.
What appears in the error output from this block?
Oh, then I don't know how to make that method correctly
return pageEscaped;
}
public String getPage(InputStream is) throws IOException {
As a matter of general guidance, public methods often better handle excep=
tions
than pass them upstream. Certainly they should log the error before ha=
ndling
it, and if it must rethrow, often it's better to wrap the low-level excep=
tion
('IOException') in an application-specific exception ('MyAppException').
There are use cases for rethrowing the low-level exception. It depends=
on the
contract for the method - whether it's a low-level method itself.
Oh, I just thought I was better to catch all exceptions in GUI
BufferedReader br = new BufferedReader(new InputStrea=
mReader
(is));
String line = "";
This initialization is never used, so don't initialize 'line' to this val=
ue.
Oh, how is it then initialized in the while?
StringBuilder sb = new StringBuilder();
while ((line = br.readLine()) != null) {
sb.append(line+'\n');
It's a bit strange that you use '\n' as the line terminator when it's app=
arent
from your code example that you're using Windows.
I think I just have to use a line terminator
System.out.println(line);
}
return sb.toString();
}
An alternative formulation for the loop that restricts the scope of 'line=
' to
just the loop is:
for ( String line; (line = br.readLine()) != null; )
{
sb.append( line + System.getProperty( "line.separator" );
System.out.println(line); // Why?
}