Groovy htmlunit getFirstByXPath devolviendo una pregunta nula + OCR

I have had a few issues with HtmlUnit returning nulls lately and am looking for guidance. each of my results for grabbing the first row of a website have returned null. I am wondering if someone can

A) explain why they might be returning null

B) explain better ways (if there are some) to go about getting the information

Here is my current code (URL is in the source):

client = new WebClient(BrowserVersion.FIREFOX_3)
client.javaScriptEnabled = false

def url = "http://www.hidemyass.com/proxy-list/"

page = client.getPage(url)

IpAddress = page.getFirstByXPath("//html/body/div/div/form/table/tbody/tr/td[2]").getValue()
println "IP Address is: $data"          //returns null

//Port_Number is an Image

Country = page.getFirstByXPath("//html/body/div/div/form/table/tbody/tr/td[4][@class='country']/@rel").getValue()
println "Country abbreviation is: $Country"

//differentiate speed and connection by name of gif?

Type = page.getFirstByXPath("//html/body/div/div/form/table/tbody/tr/td[7]").getValue()
println "Proxy type is: $Type"

Anonymity = page.getFirstByXPath("//html/body/div/div/form/table/tbody/tr/td[8]").getValue()
println "Anonymity Level is: $Anonymity"

client.closeAllWindows()

Right now all of my XPaths return null and .getValue() obviously doesn't work on null.

I also have questions as to what I should do about the PORT since it is an image? Is there a better alternative than downloading it and attempting to solve it by OCR?

Nota al margen

There is no significance in this site, I was just looking for a site that I could practice scraping on (the last one I ran into issues of fragment identities and couldn't get an answer to: HtmlUnit getByXpath devuelve nulo y Identidades HtmlUnit y Fragment )

preguntado el 08 de enero de 11 a las 18:01

1 Respuestas

It looks like your xpath query is incorrect. Based on the url provided in the code sample the form element should be removed from the search path.

texto alternativo

Here is an xpath query that will be less prone to breaking when the layout of the page changes.

//table[@id='proxylist-table']/tbody/tr/td[2]

As far as the port number goes The author of that page must have wanted that portion of the data to not be scraped for some reason. Doing OCR might be your best option.

However, one thing you could do is look at the size of the image that is returned to guess the port number. For example I've noticed that images that display port 80 all have a content length of 406 or 411. Port 8080 are either 402 or 409. There are two different sizes to the images to blend in with the row color. If the Url ends in a 1 it will have a white back ground if it ends in 0 it will have a light grey back ground and always be a few bytes larger. There are obvious drawbacks to this approach but it may work.

Respondido el 09 de enero de 11 a las 05:01

I used firebug to get my XPaths which included form (I assumed it would be correct). However, even removing formulario from the XPath returns null. - EmpezandoGroovy

Thank you for the response Aaron, it made me recheck my XPath which I have now changed to: /html/body//table/tbody/tr/td[2] which seems to give me the correct information for that value. That part is solved (for now), so the only things left are my other questions - EmpezandoGroovy

@StartingGroovy - I updated with a better XPath query and added an answer for your question about the port number being an image. - Aaron Carlson

Thank you for your answers, much appreciated :) - EmpezandoGroovy

No es la respuesta que estás buscando? Examinar otras preguntas etiquetadas or haz tu propia pregunta.