Skip to content

Commit

Permalink
Merge pull request #33056 from vespa-engine/logstash-plugin-tests
Browse files Browse the repository at this point in the history
[logstash-plugins] added unit tests and smoke integration tests
  • Loading branch information
radu-gheorghe authored Dec 20, 2024
2 parents af1b197 + 6fdfaa1 commit 801d6d2
Show file tree
Hide file tree
Showing 14 changed files with 1,012 additions and 14 deletions.
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
## 0.3.0
added retry+backoff logic

## 0.2.0
Added support for mTLS certificates, selector, page_size, backend_concurrency, timeout, from_timestamp, and to_timestamp

Expand Down
13 changes: 13 additions & 0 deletions integration/logstash-plugins/logstash-input-vespa/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,13 @@ export LOGSTASH_SOURCE=1
bundle exec rspec
```

To run integration tests, you'll need to have a Vespa instance running + Logstash installed. Check out the `integration-test` directory for more information.

```
cd integration-test
./run_tests.sh
```

## Usage

Minimal Logstash config example:
Expand Down Expand Up @@ -62,6 +69,12 @@ input {
# HTTP request timeout
timeout => 180
# maximum retries for failed HTTP requests
max_retries => 3
# delay in seconds for the first retry attempt. We double this delay for each subsequent retry.
retry_delay => 1
# lower timestamp bound (microseconds since epoch)
from_timestamp => 1600000000000000
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
input {
vespa {
vespa_url => "http://localhost:8080"
cluster => "VESPA_CLUSTER"
selection => "id.specific == 'TEST_DOC_ID'"
}
}

output {
file {
path => "/tmp/output.json"
codec => json_lines
}
stdout { codec => rubydebug }
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
#!/bin/bash

# Configuration
LOGSTASH_HOME="/opt/logstash/logstash-current"
VESPA_URL="http://localhost:8080"
VESPA_CLUSTER="used_car"

# extract the plugin version from the gemspec
PLUGIN_VERSION=$(grep version ../logstash-input-vespa.gemspec | awk -F"'" '{print $2}')
if [ -z "$PLUGIN_VERSION" ]; then
echo -e "${RED}Error: Failed to extract plugin version${NC}"
exit 1
else
echo "Plugin version: $PLUGIN_VERSION"
fi

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m'

command_exists() {
command -v "$1" >/dev/null 2>&1
}

# Check prerequisites
if [ ! -d "$LOGSTASH_HOME" ]; then
echo -e "${RED}Error: Logstash not found at $LOGSTASH_HOME${NC}"
exit 1
fi

if ! command_exists curl; then
echo -e "${RED}Error: curl is required but not installed${NC}"
exit 1
fi

# Build and install plugin
echo "Building plugin..."
cd ..
gem build logstash-input-vespa.gemspec
cd integration-test

echo "Installing plugin..."
$LOGSTASH_HOME/bin/logstash-plugin install --no-verify ../logstash-input-vespa-${PLUGIN_VERSION}.gem

# Wait for Vespa to be ready
echo "Checking Vespa availability..."
max_attempts=30

attempt=1
while ! curl --output /dev/null --silent --fail "$VESPA_URL"; do
if [ $attempt -eq $max_attempts ]; then
echo -e "${RED}Error: Vespa not available after $max_attempts attempts${NC}"
exit 1
fi
printf '.'
sleep 1
attempt=$((attempt + 1))
done
echo -e "${GREEN}Vespa is ready${NC}"

# Run test cases
echo "Running tests..."
test_count=0
failed_count=0

run_test() {
local test_name=$1
local doc_id=$2
local doc_content=$3

test_count=$((test_count + 1))
echo "Test: $test_name"

# Feed document to Vespa
curl -X POST -H "Content-Type:application/json" --data "$doc_content" \
"$VESPA_URL/document/v1/cars/$VESPA_CLUSTER/docid/$doc_id"

# Create config file with actual ID
sed "s/TEST_DOC_ID/$doc_id/" config/logstash_input_test.conf > config/logstash_input_test_with_id.conf

# fill in the Vespa cluster
sed "s/VESPA_CLUSTER/$VESPA_CLUSTER/" config/logstash_input_test_with_id.conf > config/logstash_input_test_with_id2.conf
mv config/logstash_input_test_with_id2.conf config/logstash_input_test_with_id.conf

# Run Logstash
$LOGSTASH_HOME/bin/logstash -f $(pwd)/config/logstash_input_test_with_id.conf

# Check output file
OUTPUT_FILE="/tmp/output.json"
if [ -f "$OUTPUT_FILE" ] && grep -q "$doc_id" "$OUTPUT_FILE"; then
echo -e "${GREEN}✓ Test passed${NC}"
else
echo -e "${RED}✗ Test failed - Document not found in output${NC}"
failed_count=$((failed_count + 1))
fi

# Clean up
rm -f "$OUTPUT_FILE"
}

# Create output directory
mkdir -p data

# ID will be like test_1234567890
ID="test_$(date +%s)"

# Run tests
run_test "Simple document" "$ID" '{
"fields": {
"id": "'$ID'"
}
}'

# Print summary
echo "Tests completed: $test_count, Failed: $failed_count"
exit $failed_count
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
require "logstash/devutils/rspec/spec_helper"
require "logstash/inputs/vespa"
require "webmock/rspec"
require 'tempfile'
require 'openssl'

describe LogStash::Inputs::Vespa do
let(:config) do
Expand Down Expand Up @@ -113,4 +115,157 @@
end
end
end

describe "#register" do
let(:temp_cert) do
file = Tempfile.new(['cert', '.pem'])
# Create a self-signed certificate for testing
key = OpenSSL::PKey::RSA.new(2048)
cert = OpenSSL::X509::Certificate.new
cert.version = 2
cert.serial = 1
cert.subject = OpenSSL::X509::Name.parse("/CN=Test")
cert.issuer = cert.subject
cert.public_key = key.public_key
cert.not_before = Time.now
cert.not_after = Time.now + 3600

# Sign the certificate
cert.sign(key, OpenSSL::Digest::SHA256.new)

file.write(cert.to_pem)
file.close
file
end

let(:temp_key) do
file = Tempfile.new(['key', '.pem'])
# Create a valid RSA key for testing
key = OpenSSL::PKey::RSA.new(2048)
file.write(key.to_pem)
file.close
file
end

after do
temp_cert.unlink
temp_key.unlink
end

it "raises error when only client_cert is provided" do
invalid_config = config.merge({"client_cert" => temp_cert.path})
plugin = described_class.new(invalid_config)

expect { plugin.register }.to raise_error(LogStash::ConfigurationError,
"Both client_cert and client_key must be set, you can't have just one")
end

it "raises error when only client_key is provided" do
invalid_config = config.merge({"client_key" => temp_key.path})
plugin = described_class.new(invalid_config)

expect { plugin.register }.to raise_error(LogStash::ConfigurationError,
"Both client_cert and client_key must be set, you can't have just one")
end

it "correctly sets up URI parameters" do
full_config = config.merge({
"selection" => "true",
"from_timestamp" => 1234567890,
"to_timestamp" => 2234567890,
"page_size" => 50,

"backend_concurrency" => 2,
"timeout" => 120
})

plugin = described_class.new(full_config)
plugin.register

# Access the private @uri_params using send
uri_params = plugin.send(:instance_variable_get, :@uri_params)
expect(uri_params[:selection]).to eq("true")
expect(uri_params[:fromTimestamp]).to eq(1234567890)
expect(uri_params[:toTimestamp]).to eq(2234567890)
expect(uri_params[:wantedDocumentCount]).to eq(50)
expect(uri_params[:concurrency]).to eq(2)
expect(uri_params[:timeout]).to eq(120)
end
end

describe "#parse_response" do
it "handles malformed JSON responses" do
response = double("response", :body => "invalid json{")
result = plugin.parse_response(response)
expect(result).to be_nil
end

it "successfully parses valid JSON responses" do
valid_json = {
"documents" => [{"id" => "doc1"}],
"documentCount" => 1
}.to_json
response = double("response", :body => valid_json)

result = plugin.parse_response(response)
expect(result["documentCount"]).to eq(1)
expect(result["documents"]).to be_an(Array)
end
end

describe "#process_documents" do
it "creates events with correct decoration" do
documents = [
{"id" => "doc1", "fields" => {"field1" => "value1"}},
{"id" => "doc2", "fields" => {"field1" => "value2"}}
]

# Test that decoration is applied
expect(plugin).to receive(:decorate).twice

plugin.process_documents(documents, queue)
expect(queue.size).to eq(2)

event1 = queue.pop
expect(event1.get("id")).to eq("doc1")
expect(event1.get("fields")["field1"]).to eq("value1")

event2 = queue.pop
expect(event2.get("id")).to eq("doc2")
expect(event2.get("fields")["field1"]).to eq("value2")
end
end

describe "#stop" do
it "sets stopping flag" do
plugin.stop
expect(plugin.instance_variable_get(:@stopping)).to be true
end

it "interrupts running visit operation" do
request_made = Queue.new # Use a Queue for thread synchronization

# Setup a response that would normally continue
stub_request(:get, "#{base_uri}?#{uri_params}")
.to_return(status: 200, body: {
documents: [{"id" => "doc1"}],
documentCount: 1,
continuation: "token"
}.to_json)
.with { |req| request_made.push(true); true } # Signal when request is made

# Run in a separate thread
thread = Thread.new { plugin.run(queue) }

# Wait for the first request to be made
request_made.pop

# Now we know the first request has been made, stop the plugin
plugin.stop
thread.join

# Should only make one request despite having a continuation token
expect(a_request(:get, "#{base_uri}?#{uri_params}")).to have_been_made.once
end
end
end
14 changes: 13 additions & 1 deletion integration/logstash-plugins/logstash-output-vespa/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ If you're developing the plugin, you'll want to do something like:
```
# build the gem
./gradlew gem
# run tests
./gradlew test
# install it as a Logstash plugin
/opt/logstash/bin/logstash-plugin install /path/to/logstash-output-vespa/logstash-output-vespa_feed-0.4.0.gem
/opt/logstash/bin/logstash-plugin install /path/to/logstash-output-vespa/logstash-output-vespa_feed-0.5.2.gem
# profit
/opt/logstash/bin/logstash
```
Expand All @@ -24,6 +26,16 @@ Some more good info about Logstash Java plugins can be found [here](https://www.
It looks like the JVM options from [here](https://github.com/logstash-plugins/.ci/blob/main/dockerjdk17.env)
are useful to make JRuby's `bundle install` work.

### Integration tests
To run integration tests, you'll need to have a Vespa instance running + Logstash installed. Check out the `integration-test` directory for more information.

```
cd integration-test
./run_tests.sh
```

### Publishing the gem

Note to self: for some reason, `bundle exec rake publish_gem` fails, but `gem push logstash-output-vespa_feed-$VERSION.gem`
does the trick.

Expand Down
2 changes: 1 addition & 1 deletion integration/logstash-plugins/logstash-output-vespa/VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.5.1
0.5.2
Loading

0 comments on commit 801d6d2

Please sign in to comment.