Re: HTTPUrlConnection does not download the whole page
The87Boy wrote:
I have a problem with this code, as you can see in print, where it
prints the error in the page's code:
What error? Why not copy and paste the error message in your post so that we
can actually have a prayer of helping you?
public void print(String link) {
String page = this.getPage(link);
You don't need to, and shouldn't, prefix member method calls with "this.".
For one thing, it's misleading in the presence of overridden methods, or if
'this' class doesn't override the method.
Lighten up on the indent width! Four spaces is about the maximum per indent
level that's suitable for Usenet posts.
// Here I can see the error as it prints the error in the
page's code
What error?
System.out.println(page);
System.err.println("1234567890+");
}
public String getPage(String link) {
String pageEscaped = "";
try {
URL url = new URL(link);
// Open the Connection
HttpURLConnection conn = (HttpURLConnection)
url.openConnection();
// Set the information
conn.setRequestProperty("user_agent", "Mozilla/5.0
(Windows; U; Windows NT 6.0; da-DK; rv:1.9.1.4) Gecko/20091016 Firefox/
3.5.4 (.NET CLR 3.5.30729)");
conn.setRequestProperty("max_redirects", "0");
conn.setRequestProperty("timeout", "300");
conn.setRequestMethod("GET");
conn.setDoOutput(true);
// Connect
conn.connect();
// Get the Status-Code and add it to the HashMap
int statusCode = conn.getResponseCode();
String page = this.getPage(conn.getInputStream());
pageEscaped = StringEscapeUtils.unescapeHtml(page);
conn.disconnect();
} catch (IOException e) {System.err.println(e.getCause
());System.err.println(e.getMessage());}
You problem stems at least in part that you continue blithely along pretending
to process the URL after you catch an exception.
What appears in the error output from this block?
return pageEscaped;
}
public String getPage(InputStream is) throws IOException {
As a matter of general guidance, public methods often better handle exceptions
than pass them upstream. Certainly they should log the error before handling
it, and if it must rethrow, often it's better to wrap the low-level exception
('IOException') in an application-specific exception ('MyAppException').
There are use cases for rethrowing the low-level exception. It depends on the
contract for the method - whether it's a low-level method itself.
BufferedReader br = new BufferedReader(new InputStreamReader
(is));
String line = "";
This initialization is never used, so don't initialize 'line' to this value.
StringBuilder sb = new StringBuilder();
while ((line = br.readLine()) != null) {
sb.append(line+'\n');
It's a bit strange that you use '\n' as the line terminator when it's apparent
from your code example that you're using Windows.
System.out.println(line);
}
return sb.toString();
}
An alternative formulation for the loop that restricts the scope of 'line' to
just the loop is:
for ( String line; (line = br.readLine()) != null; )
{
sb.append( line + System.getProperty( "line.separator" );
System.out.println(line); // Why?
}
Check out
<http://sscce.org/>
--
Lew