Skip to content

(#2) Make indexer-elastic work with ES8. #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
blairlearn wants to merge 1 commit into
base: nci-develop
Choose a base branch
from
Draft

(#2) Make indexer-elastic work with ES8. #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/master-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ on:
branches: [master]
pull_request:
types: [opened, synchronize, reopened]
branches: [master]
branches: [master, nci-develop]
jobs:
javadoc:
strategy:
Expand Down Expand Up @@ -104,6 +104,8 @@ jobs:
- 'src/testresources/**'
plugins:
- 'src/plugin/**'
indexer_plugins:
- 'src/plugin/indexer-*/**'
buildconf:
- 'build.xml'
- 'ivy/ivy.xml'
Expand All @@ -120,6 +122,10 @@ jobs:
- name: test plugins
if: ${{ steps.filter.outputs.plugins == 'true' && steps.filter.outputs.core == 'false' && steps.filter.outputs.buildconf == 'false' }}
run: ant clean test-plugins -buildfile build.xml
# run indexer integration tests when indexer plugin files change (Docker required, ubuntu-latest only)
- name: test indexer integration
if: ${{ steps.filter.outputs.indexer_plugins == 'true' && matrix.os == 'ubuntu-latest' }}
run: ant clean test-indexer-integration -buildfile build.xml
- name: Check for test results
id: check_tests
if: always() && matrix.os == 'ubuntu-latest'
Expand All @@ -139,4 +145,4 @@ jobs:
path: |
./build/test/TEST-*.xml
./build/**/test/TEST-*.xml
retention-days: 1
retention-days: 1
16 changes: 3 additions & 13 deletions LICENSE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ This product bundles some components that are also licensed under
the Apache License Version 2.0:


co.elastic.clients:elasticsearch-java
com.amazonaws:aws-java-sdk-cloudsearch
com.amazonaws:aws-java-sdk-core
com.beust:jcommander
Expand Down Expand Up @@ -494,20 +495,7 @@ org.eclipse.jetty.toolchain:jetty-servlet-api
org.eclipse.jetty.websocket:websocket-api
org.eclipse.jetty.websocket:websocket-client
org.eclipse.jetty.websocket:websocket-common
org.elasticsearch:elasticsearch
org.elasticsearch:elasticsearch-cli
org.elasticsearch:elasticsearch-core
org.elasticsearch:elasticsearch-geo
org.elasticsearch:elasticsearch-secure-sm
org.elasticsearch:elasticsearch-x-content
org.elasticsearch:jna
org.elasticsearch.client:elasticsearch-rest-client
org.elasticsearch.client:elasticsearch-rest-high-level-client
org.elasticsearch.plugin:aggs-matrix-stats-client
org.elasticsearch.plugin:lang-mustache-client
org.elasticsearch.plugin:mapper-extras-client
org.elasticsearch.plugin:parent-join-client
org.elasticsearch.plugin:rank-eval-client
org.gagravarr:vorbis-java-core
org.gagravarr:vorbis-java-tika
org.jetbrains:annotations
Expand Down Expand Up @@ -721,6 +709,8 @@ Eclipse Public License - Version 2.0

(licenses-binary/LICENSE-eclipse-public-license---version-2.0.txt)

jakarta.json:jakarta.json-api
org.eclipse.parsson:parsson
org.eclipse.jetty:jetty-http
org.eclipse.jetty:jetty-io
org.eclipse.jetty:jetty-security
Expand Down
53 changes: 13 additions & 40 deletions NOTICE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,10 @@ AOP alliance (http://aopalliance.sourceforge.net)
- license: Public Domain
(licenses-binary/LICENSE-public-domain.txt)

# co.elastic.clients:elasticsearch-java
Elasticsearch Java API Client (https://github.com/elastic/elasticsearch-java/)
- license: The Apache Software License, Version 2.0

# com.adobe.xmp:xmpcore
Adobe XMPCore (https://www.adobe.com/devnet/xmp/library/eula-xmp-library-java.html)
- license: The BSD 3-Clause License (BSD3)
Expand Down Expand Up @@ -985,6 +989,11 @@ Jakarta Annotations API (https://projects.eclipse.org/projects/ee4j.ca)
- license: EPL 2.0
(licenses-binary/LICENSE-epl-2.0.txt)

# jakarta.json:jakarta.json-api
Jakarta JSON Processing API (https://github.com/eclipse-ee4j/jsonp)
- license: Eclipse Public License 2.0
(licenses-binary/LICENSE-epl-2.0.txt)

# jakarta.jws:jakarta.jws-api
Jakarta Web Services Metadata API (https://github.com/eclipse-ee4j/jws-api)
- license: Eclipse Distribution License - v 1.0
Expand Down Expand Up @@ -1254,50 +1263,14 @@ Jetty :: Websocket :: Client
Jetty :: Websocket :: Common
- license: Apache Software License - Version 2.0

# org.elasticsearch:elasticsearch
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch:elasticsearch-cli
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch:elasticsearch-core
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch:elasticsearch-geo
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch:elasticsearch-secure-sm
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch:elasticsearch-x-content
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch:jna
Elastic JNA Distribution (https://github.com/java-native-access/jna)
- license: Apache License, Version 2.0
# org.eclipse.parsson:parsson
Eclipse Parsson
- license: Eclipse Public License 2.0
(licenses-binary/LICENSE-epl-2.0.txt)

# org.elasticsearch.client:elasticsearch-rest-client
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch.client:elasticsearch-rest-high-level-client
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0

# org.elasticsearch.plugin:aggs-matrix-stats-client
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch.plugin:lang-mustache-client
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch.plugin:mapper-extras-client
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch.plugin:parent-join-client
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0
# org.elasticsearch.plugin:rank-eval-client
The Apache Software License, Version 2.0 (https://github.com/elastic/elasticsearch)
- license: The Apache Software License, Version 2.0

# org.fusesource.leveldbjni:leveldbjni-all
${project.artifactId}
Expand Down
4 changes: 4 additions & 0 deletions build.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -534,6 +534,10 @@
<ant dir="src/plugin" target="test-single" inheritAll="false"/>
</target>

<target name="test-indexer-integration" depends="resolve-test, compile, compile-core-test, job" description="--> run indexer plugin integration tests (Testcontainers)">
<ant dir="src/plugin" target="test-indexer-integration" inheritAll="false"/>
</target>

<target name="nightly" depends="test, tar-src, zip-src" description="--> run the nightly target build">
</target>

Expand Down
19 changes: 11 additions & 8 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,16 +42,19 @@ WORKDIR /root/

# Install dependencies
RUN apk update
RUN apk --no-cache add apache-ant bash git openjdk11 supervisor
RUN apk --no-cache add apache-ant bash git openjdk17 supervisor

# Establish environment variables
RUN echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk' >> $HOME/.bashrc
RUN echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk' >> $HOME/.ashrc
ENV JAVA_HOME='/usr/lib/jvm/java-11-openjdk'
RUN echo 'export JAVA_HOME=/usr/lib/jvm/java-17-openjdk' >> $HOME/.bashrc
RUN echo 'export JAVA_HOME=/usr/lib/jvm/java-17-openjdk' >> $HOME/.ashrc
RUN echo 'export NUTCH_JAVA_HOME=/usr/lib/jvm/java-17-openjdk' >> $HOME/.bashrc
RUN echo 'export NUTCH_JAVA_HOME=/usr/lib/jvm/java-17-openjdk' >> $HOME/.ashrc
ENV JAVA_HOME='/usr/lib/jvm/java-17-openjdk'
ENV NUTCH_JAVA_HOME='/usr/lib/jvm/java-17-openjdk'
ENV NUTCH_HOME='/root/nutch_source/runtime/local'

# Checkout and build the Nutch master branch (1.x)
RUN git clone https://github.com/apache/nutch.git nutch_source && \
# Clone from our fork.
RUN git clone --single-branch --branch v1.22-with-es8 https://github.com/nciocpl/nutch.git nutch_source && \
cd nutch_source && \
ant runtime && \
rm -rf build/ && \
Expand All @@ -78,7 +81,7 @@ ENV SERVER_HOST=$SERVER_HOST
RUN mkdir -p /var/log/supervisord
COPY ./config/supervisord_startserver.conf /etc/supervisord.conf

# Expose port for server which can only be accessed if
# Expose port for server which can only be accessed if
# the same port is published when the container is run.
EXPOSE $SERVER_PORT

Expand All @@ -105,7 +108,7 @@ RUN git clone https://github.com/apache/nutch-webapp.git nutch_webapp && \
RUN mkdir -p /var/log/supervisord
COPY ./config/supervisord_startserver_webapp.conf /etc/supervisord.conf

# Expose ports for server and webapp, these can only be accessed if
# Expose ports for server and webapp, these can only be accessed if
# the same ports are published when the container is run.
EXPOSE $SERVER_PORT
EXPOSE $WEBAPP_PORT
Expand Down
70 changes: 57 additions & 13 deletions ivy/ivy.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@
</publications>

<dependencies>
<dependency org="org.apache.logging.log4j" name="log4j-api" rev="2.25.2" conf="*->master" />
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.25.2" conf="*->master" />
<dependency org="org.apache.logging.log4j" name="log4j-slf4j2-impl" rev="2.25.2" conf="*->master" />
<dependency org="org.apache.logging.log4j" name="log4j-api" rev="2.25.4" conf="*->master" />
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.25.4" conf="*->master" />
<dependency org="org.apache.logging.log4j" name="log4j-slf4j2-impl" rev="2.25.4" conf="*->master" />
<dependency org="org.slf4j" name="slf4j-api" rev="2.0.17" conf="*->master" />

<dependency org="org.apache.commons" name="commons-lang3" rev="3.20.0" conf="*->default" />
Expand Down Expand Up @@ -74,7 +74,14 @@
</dependency>
<!-- End of Hadoop Dependencies -->

<dependency org="org.apache.tika" name="tika-core" rev="3.2.3" conf="*->default"/>
<!-- Security overrides for Hadoop-managed dependencies. -->
<dependency org="org.apache.commons" name="commons-configuration2" rev="2.15.0" conf="*->default" force="true" />
<dependency org="org.apache.zookeeper" name="zookeeper" rev="3.8.6" conf="*->default" force="true" />
<dependency org="org.apache.zookeeper" name="zookeeper-jute" rev="3.8.6" conf="*->default" force="true" />
<dependency org="com.nimbusds" name="nimbus-jose-jwt" rev="9.37.4" conf="*->default" force="true" />
<dependency org="org.bouncycastle" name="bcprov-jdk18on" rev="1.84" conf="*->default" force="true" />

<dependency org="org.apache.tika" name="tika-core" rev="3.3.1" conf="*->default"/>

<dependency org="xml-apis" name="xml-apis" rev="1.4.01" /><!-- force this version as it is required by Tika -->
<dependency org="xerces" name="xercesImpl" rev="2.12.2" />
Expand All @@ -90,15 +97,16 @@
<exclude module="hadoop-client" />
</dependency>

<dependency org="org.apache.cxf" name="cxf-rt-frontend-jaxws" rev="3.6.9" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-frontend-jaxrs" rev="3.6.9" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-transports-http" rev="3.6.9" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-transports-http-jetty" rev="3.6.9" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-rs-client" rev="3.6.9" conf="test->default" />
<dependency org="com.fasterxml.jackson.core" name="jackson-databind" rev="2.18.5" conf="*->default" />
<dependency org="com.fasterxml.jackson.core" name="jackson-annotations" rev="2.18.5" conf="*->default" />
<dependency org="com.fasterxml.jackson.dataformat" name="jackson-dataformat-cbor" rev="2.18.5" conf="*->default" />
<dependency org="com.fasterxml.jackson.jaxrs" name="jackson-jaxrs-json-provider" rev="2.18.5" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-frontend-jaxws" rev="3.6.11" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-frontend-jaxrs" rev="3.6.11" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-transports-http" rev="3.6.11" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-transports-http-jetty" rev="3.6.11" conf="*->default" />
<dependency org="org.apache.cxf" name="cxf-rt-rs-client" rev="3.6.11" conf="test->default" />
<dependency org="com.fasterxml.jackson.core" name="jackson-core" rev="2.18.6" conf="*->default" />
<dependency org="com.fasterxml.jackson.core" name="jackson-databind" rev="2.18.6" conf="*->default" />
<dependency org="com.fasterxml.jackson.core" name="jackson-annotations" rev="2.18.6" conf="*->default" />
<dependency org="com.fasterxml.jackson.dataformat" name="jackson-dataformat-cbor" rev="2.18.6" conf="*->default" />
<dependency org="com.fasterxml.jackson.jaxrs" name="jackson-jaxrs-json-provider" rev="2.18.6" conf="*->default" />

<!-- WARC artifacts needed -->
<dependency org="org.netpreserve.commons" name="webarchive-commons" rev="3.0.2" conf="*->default">
Expand Down Expand Up @@ -126,6 +134,10 @@
<dependency org="org.mockito" name="mockito-core" rev="5.18.0" conf="test->default"/>
<dependency org="org.mockito" name="mockito-junit-jupiter" rev="5.18.0" conf="test->default"/>

<!-- Testcontainers for indexer plugin integration tests -->
<dependency org="org.testcontainers" name="testcontainers" rev="2.0.3" conf="test->default"/>
<dependency org="org.testcontainers" name="junit-jupiter" rev="1.21.4" conf="test->default"/>

<!-- Jetty used to serve test pages for unit tests, but is also provided as dependency of Hadoop -->
<dependency org="org.eclipse.jetty" name="jetty-server" rev="12.1.5" conf="test->default">
<exclude org="ch.qos.reload4j" module="*" />
Expand All @@ -136,6 +148,38 @@
<!--TODO refactor these to indexer-elastic-rest plugin somehow, currently doesn't resolve correctly-->
<dependency org="org.apache.httpcomponents" name="httpcore-nio" rev="4.4.16" />
<dependency org="org.apache.httpcomponents" name="httpcore" rev="4.4.16" />
<!-- Force Netty modules to a single patched version. -->
<dependency org="io.netty" name="netty-all" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-buffer" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-dns" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-haproxy" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-http" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-http2" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-memcache" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-mqtt" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-redis" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-smtp" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-socks" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-stomp" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-codec-xml" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-common" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-handler" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-handler-proxy" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-handler-ssl-ocsp" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-resolver" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-resolver-dns" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-resolver-dns-classes-macos" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-resolver-dns-native-macos" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport-classes-epoll" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport-classes-kqueue" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport-native-epoll" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport-native-kqueue" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport-native-unix-common" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport-rxtx" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport-sctp" rev="4.1.135.Final" conf="*->default" force="true" />
<dependency org="io.netty" name="netty-transport-udt" rev="4.1.135.Final" conf="*->default" force="true" />

<dependency org="de.vandermeer" name="asciitable" rev="0.3.2" />
<dependency org="org.littleshoot" name="littleproxy" rev="1.1.2" conf="test->default"/>
Expand Down
Loading
Loading