wget para clonar un sitio web, con enlaces al directorio no index.html

I would like to clone a wordpress blog with wget so that I can include it as static content in a web app I am writing.

At the moment I am using the following to clone the site:

wget -rk http://sitename.com

This is working well, but the links in the generated html point to the index.html file. I would like those links to point to the directory that contains the file.

e.g. for the page http://sitename.com/blog-post-about-cats/ wget is generating a directory "blog-post-about-cats" and putting a index.html file in there. Links to that blog post are written as "../blog-post-about-cats/index.html" where I want them to be "../blog-post-about-cats/".

I guess I want it because I think the index.html in the url looks a bit ugly and these pages are all about presentation.

Any ideas? Is this possible with wget or perhaps a different command line tool?


preguntado el 02 de febrero de 12 a las 11:02

2 Respuestas

I assume wget doesn't do this by default because your local web server might be configured to serve up index pages for directories, rather than index.html. The simplest solution is to postprocess all the fetched HTML files afterwards with a regular expression:

find -name '*.html' | xargs sed -rie 's/href="([^"]*)\/index\.html"/href="\1\/"/gi'

If the pages on this site are some other type of file such as .php files, substitute "*.php" or whatever is suitable. The function of the regular expression is to identify strings of the form href="stuff/index.html" and remove the index.html. The xargs and find are used to apply this to all pages, and the "-i" flag to sed makes it modify files in-place. The "gi" flags in the regular expression make it replace all occurrences, and be case-insensitive (since HTML is case-insensitive).

Respondido 02 Feb 12, 16:02

Thanks mate, that will do the trick. Similar to what I had come up with in the meantime, but your solutions regexp is much nicer. - James P. McGrath

Puedes dejar wget do this for you. Try using:

wget –mirror –convert-links –html-extension –wait=1 https://example.com/

Puede encontrar más detalles aquí: https://aawaara.com/post/188869436762/how-to-clone-any-website-in-the-world-and-host-it

respondido 07 nov., 19:22

No es la respuesta que estás buscando? Examinar otras preguntas etiquetadas or haz tu propia pregunta.