This adds the possibility of exporting the nutch segments to a WARC files.

From the usage point of view a couple of new command line options are available:

• -warc: enables the functionality to export into WARC files, if not specified the default JACKSON formatter is used.
• -warcSize: enable the option to define a max file size for each WARC file, if not specified a default of 1GB per file is used as recommended by the WARC ISO standard.

The usual -gzip flag can be used to enable compression on the WARC files, which allow to compress the output files.

Some changes to the default CommonCrawlDataDumper were done, essentially some changes to the Factory and to the Formats. This changes avoid creating a new instance of a CommmonCrawlFormat on each URL read from the segments.

Hi,

For some time, I have been using @momer's Selenium plugin for Nutch 2.3, which has worked wonders in AJAX crawling. Lately however, I've noticed the following issues with it:

1. It does not support HTTPS. While adding support for HTTPS into it, I realized that:
2. It does a lot of non-Selenium stuff that makes it inefficient, including making every request twice.

I've cleaned up the code(unfortunately, the update history pointing to him is gone because of the lazy copy), making sure it uses Selenium and only selenium.

So far it has the following weaknesses:

• Selenium's Wait does not appear to be properly used(also an issue in @momer's code)
• It does not fill out the WebPage fields according to snuff. I do think someone more experienced with Selenium could get this done, or I might add support in the future.
• Does not fetch robots.txt properly.

Due to these issues, I think merging is a bit premature. However, I'd like to point out that I do not think @momer's code should be merged either.

API calls documented at - https://wiki.apache.org/nutch/Nutch_1.X_RESTAPI. This pull request includes Index, Generate, Fetch, Parse, Update, InvertLinks, Dedup and Readdb jobs.

• Add extension point org.apache.nutch.net.URLExemptionFilter
• Modify FetcherThread and ParseOutputFormat to integrate new extension point
• Add extension urlfilter-ignoreexempt
• build configs modified to include new extension

Resolves https://issues.apache.org/jira/browse/NUTCH-2144

• adds an indexer plugin which writes a configurable CSV index
• works only in local mode in combination with -noCommit

CSVIndexWriter - write index as CSV file (comma separated values)
  indexer.csv.fields       : ordered list of fields (columns) in the CSV file
  indexer.csv.separator    : separator between fields (columns), default: , (U+002C, comma)
  indexer.csv.quotechar    : quote character used to quote fields containing separators or quotes, default: " (U+0022, quotation mark)
  indexer.csv.escapechar   : escape character used to escape a quote character, default: " (U+0022, quotation mark)
  indexer.csv.recordsep    : separator between records (rows) resp. documents, default: \r\n (DOS-style line breaks)
  indexer.csv.valuesep     : separator between multiple values of one field, default: | (U+007C)
  indexer.csv.maxfieldvalues : max. number of values of one field, useful for,  e.g., the anchor texts field, default: 12
  indexer.csv.maxfieldlength : max. length of a single field value in characters, default: 4096.
  indexer.csv.charset      : encoding of CSV file, default: UTF-8
  indexer.csv.header       : write CSV column headers, default: true
  indexer.csv.outpath      : output path / directory, default: csvindexwriter. 
    CAVEAT: existing output directories are removed!

With this patch now we can have many instances of the same IndexWriter class, but with different configurations. Also, we can copy, rename or remove fields of documents for every index writer individually. Besides, the parameters needed by the index writers will be into separated XML files, so them will be not into nutch-site.xml anymore.

An index writer for HBase like index writer for Solr, ElasticSearch etc. Expected HBase table description and NutchDocument to HBase mapping is read using a indexer-solr alike mapping file and write NutchDocument fields into the table of a HBase server.

TODO: Functionality to send and set kerberos authentication configuration for secure hdfs

use the headless option of firefox and chrome in protocol-selenium

the --headless option is added to firefox in version 55 or later , and in chrome in version 59 or later ... this is much better than relying on xvfb and its associates . we can add it as a property in the config file . I'm trying it on my local machine , and will create a pull request when I finish testing it

I've tested it using firefox 57.0 , gecodriver 0.19.1 and selenium 3.7.1

Important note : you need to add the following property to nutch-default.xml or nutch-site.xml for the headless option to work

Upgrade Elasticsearch and Lucene dependencies, which, in turn, requires updates to Guava and Hadoop dependencies:

• Elasticsearch 1.4.1 -> Elasticsearch 2.3.3
• Lucene 4.10.2 -> 5.5.0
• Guava 16.0.1 -> Guava 18.0
• Hadoop 2.4.0 -> 2.7.2

As described on JIRA:

This plugin allows for collecting uri links from CSS (stylesheets). This is useful for collecting parent stylesheets, fonts, and images needed to display web pages as intended.

Parsed Outlinks do not have associated anchors, and no additional text/content is parsed from the stylesheet.

OK folks, this issue addresses https://issues.apache.org/jira/browse/NUTCH-2184 by

• rebasing the NUTCH-2184v2.patch against master branch
• making the IndexerMapReduceMapper and IndexerMapReduceReducer in IndexerMapReduce code explicit so that these functions can be tested
• adding in some mrunit tests for testing the IndexerMapReduceMapper and IndexerMapReduceReducer
• removing some trivial imports which are unsed
• formatting ivy.xml which has somehow (again) become a dogs dinner
• adding default constructor to NutchIndexAction()

Any questions, then please let me know. I would really appreciate if people could pull this code and try it out within your test or local environment. Thanks, also thanks Markus for the original suggestions for tests, etc.

This is a WIP for https://issues.apache.org/jira/browse/NUTCH-2940. The work was conducted by @AzureTriple @imanzanganeh @jbsimmon @LilyPerr and @Lirongxuan1 from the 2022 USC Senior CS Capstone Program.

Most of the core build is in place. No plugin sub-projects have been implemented yet.

I intend to continue work on the code build until it is completed. I will then move on to the plugin sub-projects.

PR addresses https://issues.apache.org/jira/browse/NUTCH-2938 We could improve the performance of this plugin if we could reuse the repository connection however I am not entirely sure how to do that right now because this is done down in the Any23 layer.

Until I complete the Nutch Metrics and we agree on NUTCH-2909 I would ask that you don't look at this with too much sincerity. I'm submitting this early so anyone interested can see where I thought we could go with this one. Thanks for any feedback.

Before the change, the output file name was hard-coded to "nutch.csv". When running in distributed mode, multiple reducers would clobber each other output.

After the change, the filename is taken from the first open(cfg, name) initialization call, where name is a unique file name generated by IndexerOutputFormat, derived from hadoop FileOutputFormat. The CSV files are now named like part-r-000xx.

• apply patch contributed by @albinscode
• load configuration files from classpath and address thread-safety

Note: not ready yet:

• TODOs in code
• unit tests fail (with DOM built by tagsoup parser)
• see also open points in NUTCH-1870