Just some feedback, thanks.

This tool is really cool and promising, but I haven't seen much happen with it, I'm a little afraid that it may be dead. Comment?

A functionality to check for redirections would be useful.

What should pagetrawler do when a site is generated from api/db calls - e.g. wordpress

Firstly; it would be great if the CSV download gave you a whole site or if you could select a part of a site through a folder reference.
SEO wise would be good if you could have onpage factors listed other than page title; H1, meta description, meta keywords, on page text ratio to code, #of links with a link to open a separate file of all links into the page both external and internal.

Images, files, pages that are not linked from anywhere and could be removed from the server.

Pull metadata from on-page data formats like WP author (and make that page owner?), publish dates, etc.
Same thing for microformats. Parsing microformats found on the page could provide a lot of metadata.

The ability to specify exclusion or limitation of content – for example, exclude */tags/* – would be nice.

Make a .txt document from all the text pages, to form a master document for the rewrite. At the moment it's all cut and paste from each individual page.

Shade levels 0-3 with 0 darkest. Makes it easy to pick out top-level pages

Xenu has a column in the CSV for 'type' which quickly helps me sort
applications and images from text/html pages

I'd like to see a pagemap/visual representation of how all the pages on the site link together.

Perhaps also downloadable as a PDF :-)

Search within the text of each page for user-defined keywords and also repetitions of strings over x chars long (i.e. to see if your content is repeating itself unnecessarily)

It would be awesome if the service could visualise my site, and cluster sets of related content together. Initially, that could just be by looking at the site arcitecture, but it would be awesome if it could also search the content of pages and make educated guesses based on that.

I'd like to pull in my Google analytics, and see hits, time on page, etc.

I want page trawler to take a screen grab of every page.