Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing ClientRequestFilter.java: A New Plugin for Merging Additional Headers into Client Requests in the Authentication Filter #23380

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

SthuthiGhosh9400
Copy link
Contributor

@SthuthiGhosh9400 SthuthiGhosh9400 commented Aug 5, 2024

Description

  1. A new interface, ClientRequestFilter.java, has been introduced in the Presto SPI to customize the headers used by the Presto runtime to process queries. Some example use cases include customized authentication workflows, or enriching query attributes such as the query source.
  2. One key use case is setting extra credentials in the request headers.

Motivation and Context

#23997

Impact

This feature enables the merging of headers of any type, offering a versatile solution for header management.

Test Plan

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==
General Changes
* Add ``ClientRequestFilter.java`` interface in Presto-spi. :pr:`23380`
* Improve Request Headers in the Authentication Filter Class. :pr:`23380`

Copy link
Contributor

@tdcmeehan tdcmeehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR needs some test cases.

@steveburnett
Copy link
Contributor

Nit rephrasing of release note entries following the Order of changes in the Release Notes Guidelines.

== RELEASE NOTES ==

General Changes

* Add  `RequestModifier.java` interface in Presto-spi :pr:`23380`
* Improve Request Headers in the Authentication Filter Class :pr:`23380`

@SthuthiGhosh9400
Copy link
Contributor Author

@tdcmeehan

I have added a test case to verify setting values in the request header and committed it.
Could you have a check and confirm ?
Thanks.

presto-main/pom.xml Show resolved Hide resolved
assertEquals("CustomValue", wrappedRequest.getHeader("X-Custom-Header"));
}

abstract static class HttpServletRequestAdapter
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's already a MockHttpServletRequest, can you use that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdcmeehan
I have refactored the code to use MockHttpServletRequest. Could you check it?
Thanks.

@tdcmeehan
Copy link
Contributor

While we don't use Mockito, the stubs in the test are a little out of control. We need to reduce the size of this simple test. I have a couple of suggestions that you can research to make the testing simpler.

  • Can you look at TestHttpRemoteTask and examine how they implemented an in-memory endpoint for the purposes of testing? Perhaps you can build something similar.
  • TestingPrestoServer has all sorts of helper methods to retrieve individual classes from Presto. Perhaps you can add a new getter to retrieve the RequestModifierManager, register a simple new request modifier that just adds a dummy header, and make a client call that proves the header is added.

@SthuthiGhosh9400
Copy link
Contributor Author

  • TestingPrestoServer has all sorts of helper methods to retrieve individual classes from Presto. Perhaps you can add a new getter to retrieve the RequestModifierManager, register a simple new request modifier that just adds a dummy header, and make a client call that proves the header is added.

@tdcmeehan
I have attempted to improve the test case based on the suggestion. Could you have a check?
Thanks.

@@ -825,4 +830,25 @@ private static int driftServerPort(DriftServer server)
{
return ((DriftNettyServerTransport) server.getServerTransport()).getPort();
}

public RequestModifierManager getRequestModifierManager()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, just add a method to return the RequestModifierManager, and let users inject whatever custom modifier they wish.


public class RequestModifierManager
{
private final List<RequestModifier> requestModifiers;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be thread safe. I recommend using a CopyOnWriteArrayList.

// authentication succeeded
nextFilter.doFilter(withPrincipal(request, principal), response);
CustomHttpServletRequestWrapper wrappedRequest = withPrincipal(request, principal);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of modifying the authentication filter, this should just go in its own filter that is applied after the authentication filter.

this.customHeaders = new HashMap<>();
}

public void addHeader(String name, String value)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused

import java.util.Map;
import java.util.Optional;

public interface RequestModifier
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's call this something more descriptive, such as ClientRequestFilter.

@SthuthiGhosh9400
Copy link
Contributor Author

@tdcmeehan
I have incorporated the review comments. Could you please verify them?
Thanks.

@SthuthiGhosh9400 SthuthiGhosh9400 changed the title Introducing RequestModifier.java: A New Plugin for Modifying Request Headers in the Authentication Filter Class Introducing ClientRequestFilter.java: A New Plugin for Applying Request Headers in the Authentication Filter Class Sep 10, 2024

public class ClientRequestFilterManager
{
private final CopyOnWriteArrayList<ClientRequestFilter> clientRequestFilters;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private final CopyOnWriteArrayList<ClientRequestFilter> clientRequestFilters;
private final List<ClientRequestFilter> clientRequestFilters;

Comment on lines 24 to 54
private final CopyOnWriteArrayList<ClientRequestFilter> clientRequestFilters;
public ClientRequestFilterManager()
{
this.clientRequestFilters = new CopyOnWriteArrayList<>();
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private final CopyOnWriteArrayList<ClientRequestFilter> clientRequestFilters;
public ClientRequestFilterManager()
{
this.clientRequestFilters = new CopyOnWriteArrayList<>();
}
private final List<ClientRequestFilter> clientRequestFilters = new CopyOnWriteArrayList<>();


public List<ClientRequestFilter> getClientRequestFilters()
{
return Collections.unmodifiableList(clientRequestFilters);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return Collections.unmodifiableList(clientRequestFilters);
return ImmutableList.copyOf(clientRequestFilters);

extraHeaderValueMap.ifPresent(map -> {
for (Map.Entry<String, String> extraHeaderEntry : map.entrySet()) {
if (request.getHeader(extraHeaderEntry.getKey()) == null) {
extraHeadersMap.putIfAbsent(extraHeaderEntry.getKey(), extraHeaderEntry.getValue());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should silently not put the header if there's an existing header. I think we should consider the following:

  1. The SPI contract mandates that any potential header is known upfront.
  2. The runtime will validate that the headers that are requested to be modified match with what is actually returned.
  3. There is a blocklist of headers that the plugin should not modify (such as query ID).
  4. The runtime will validate that all registered filters only return a disjoint set of headers being added. If there is a conflict, this is a fatal error that will require an administrator to fix (but un-registering one of these plugins).
  5. If there is a conflict here, then an exception is thrown.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdcmeehan
I have gone through your comments and added a few validations in the current implementation. I can verify that with you. However, I have to know about the headers in the blocklist that should not be modified.
Could you help?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's start with the query ID, trace ID and transaction ID in PrestoHeaders.java.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have incorporated the changes mentioned in your comment. Could you please review them and share your suggestions?

Comment on lines 203 to 234
private final Map<String, String> customHeaders;

public CustomHttpServletRequestWrapper(HttpServletRequest request)
{
super(request);
this.customHeaders = new ConcurrentHashMap<>();
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private final Map<String, String> customHeaders;
public CustomHttpServletRequestWrapper(HttpServletRequest request)
{
super(request);
this.customHeaders = new ConcurrentHashMap<>();
}
private final Map<String, String> customHeaders = new ConcurrentHashMap<>();

Copy link
Contributor

@tdcmeehan tdcmeehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some feedback is still unadressed.

for (Map.Entry<String, String> extraHeaderEntry : map.entrySet()) {
String headerKey = extraHeaderEntry.getKey();
if (headersBlockList.contains(headerKey)) {
throw new RuntimeException("Modification attempt detected: The header " + headerKey + " is present in the blocked headers list.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please throw a PrestoException and introduce a new error code for this. It should be a system error.

while (originalHeaderNames.hasMoreElements()) {
headerNames.add(originalHeaderNames.nextElement());
}
return Collections.enumeration(headerNames);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use Immutable collections.

@SthuthiGhosh9400
Copy link
Contributor Author

Some feedback is still unadressed.

@tdcmeehan Could you help me identify which feedback is still pending? I believe I have addressed feedback items 2, 3, 4, and 5 from the list above.

@tdcmeehan
Copy link
Contributor

Some feedback is still unadressed.

@tdcmeehan Could you help me identify which feedback is still pending? I believe I have addressed feedback items 2, 3, 4, and 5 from the list above.

Please scroll up in this PR and examine feedback on the code that is still remaining.

@SthuthiGhosh9400
Copy link
Contributor Author

Please scroll up in this PR and examine feedback on the code that is still remaining.

Yeah. @tdcmeehan I have incorporated all feedback items now. Kindly review them and share your suggestions?

Thanks.

Comment on lines 215 to 113
ClientRequestFilter customModifier = new ClientRequestFilter()
{
@Override
public List<String> getHeaderNames()
{
return Collections.singletonList("X-Custom-Header");
}
@Override
public <T> Optional<Map<String, String>> getExtraHeaders(T additionalInfo)
{
Map<String, String> headers = new HashMap<>();
headers.put("X-Custom-Header", "CustomValue_1");
return Optional.of(headers);
}
};

ClientRequestFilter customModifierConflict = new ClientRequestFilter()
{
@Override
public List<String> getHeaderNames()
{
return Collections.singletonList("X-Custom-Header");
}
@Override
public <T> Optional<Map<String, String>> getExtraHeaders(T additionalInfo)
{
Map<String, String> headers = new HashMap<>();
headers.put("X-Custom-Header", "CustomValue_2");
return Optional.of(headers);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reduce duplicate code. Consider an inner helper class.

}

static class ConcreteHttpServletRequest
extends MockHttpServletRequest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not modify MockHttpServletRequest directly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ConcreteHttpServletRequest, I am storing additional headers in the customHeaders map and overriding the getHeaders method to manage them. This functionality isn't natively provided by MockHttpServletRequest, so extending it makes sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just add a new constructor to MockHttpServletRequest that overrides the headers.

@SthuthiGhosh9400
Copy link
Contributor Author

@tdcmeehan
I have incorporated the review comments. Kindly review them.
If it looks good, I can squash the commits and edit the commit message according to the guidelines.

Copy link
Contributor

@tdcmeehan tdcmeehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please refactor tests to use the code under test.

for (Map.Entry<String, String> extraHeaderEntry : map.entrySet()) {
String headerKey = extraHeaderEntry.getKey();
if (headersBlockList.contains(headerKey)) {
throw new PrestoException(HEADER_MODIFICATION_ATTEMPT, "Modification attempt detected: The header " + headerKey + " is present in the blocked headers list.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of mentioning a "blocked headers list", just list out the headers that are not allowed to be modified.

throw new PrestoException(HEADER_MODIFICATION_ATTEMPT, "Modification attempt detected: The header " + headerKey + " is present in the blocked headers list.");
}
if (globallyAddedHeaders.contains(headerKey)) {
throw new RuntimeException("Header conflict detected: " + headerKey + " already added by another filter.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not throw RuntimeException, throw PrestoException.

for (Map.Entry<String, String> extraHeaderEntry : map.entrySet()) {
String headerKey = extraHeaderEntry.getKey();
if (headersBlockList.contains(headerKey)) {
throw new RuntimeException("Modification attempt detected: The header " + headerKey + " is present in the blocked headers list.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we duplicating code here? I am very confused about how this test works.

@SthuthiGhosh9400
Copy link
Contributor Author

@tdcmeehan
I have refactored the test cases and used a separate method for adding headers to the request in the Authentication filter.
Could you review it and share your suggestions?

@tdcmeehan
Copy link
Contributor

Please squash commits and ensure the commit format follows our guidelines in contributing.md.

Copy link
Contributor

@tdcmeehan tdcmeehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there. Please also add documentation for this plugin to https://github.com/prestodb/presto/tree/master/presto-docs/src/main/sphinx/develop

Comment on lines 161 to 162
ClientRequestFilterContext context = new ClientRequestFilterContext(principal);
Optional<Map<String, String>> extraHeaderValueMap = requestFilter.getExtraHeaders(context);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I realize now that the principal is per request. Context objects typically are used to create the plugin. So please revert this, and just add the principal as a plain old parameter to getExtraHeaders, no need for fancy parameterization.

Comment on lines 185 to 181
Map<String, String> extraHeadersMap = extraHeadersMapBuilder.build();
return new ModifiedHttpServletRequest(request, extraHeadersMapBuilder.build());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Map<String, String> extraHeadersMap = extraHeadersMapBuilder.build();
return new ModifiedHttpServletRequest(request, extraHeadersMapBuilder.build());
return new ModifiedHttpServletRequest(request, extraHeadersMapBuilder.build());

Remove duplicate build.

Comment on lines 167 to 179
return new ClientRequestFilter() {
@Override
public List<String> getHeaderNames()
{
return ImmutableList.of(headerName);
}

@Override
public Optional<Map<String, String>> getExtraHeaders(ClientRequestFilterContext context)
{
return Optional.of(ImmutableMap.of(headerName, headerValue));
}
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this an inner class.

Comment on lines 6 to 7
Presto facilitates seamless integration of additional headers into client requests using the
Client Request Filter plugin via the AuthenticationFilter class, enabling efficient and dynamic header management.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds AI generated. Please keep the documentation factual and concise. Example, instead of"Efficient and dynamic header management", say, "allowing operators of Presto to customize the headers used for query processing by the Presto runtime." "Presto facilitates the seamless integration of additional headers" is another example. "Seamless" has no meaning in this context.

Just say something like "Presto allows operators to customize the headers used by the Presto runtime to process queries. Some example use cases include customized authentication workflows, or enriching query attributes such as the query source." Please use original wording, think about it, and avoid tools to write this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

Comment on lines 12 to 13
The ``ClientRequestFilterFactory`` is responsible for creating instances of ``ClientRequestFilter``. It also defines
the name of the filter, which is subsequently used as the header name in client requests.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true? The name of the filter is the header name in the client request?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected.


After installing a plugin that implements ``ClientRequestFilterFactory`` on the coordinator, the ``AuthenticationFilter`` class passes the ``principal`` object to the request filter, which returns the header values as a map.

Presto uses the name of the request filter to determine whether the header is present in the blocklist.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What blocklist? Mention the headers which are not allowed to be overidden. Link to the Java source of headers that may be overidden.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.


public interface ClientRequestFilter
{
List<String> getHeaderNames();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
List<String> getHeaderNames();
Set<String> getExtraHeaderKeys();

{
List<String> getHeaderNames();

Optional<Map<String, String>> getExtraHeaders(Principal principal);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can it just return empty map and we don't return optional?

Comment on lines 14 to 15
``ClientRequestFilter`` interface provides two methods: ``getExtraHeaders()``,
which returns header values as key-value pairs in a map, and ``getHeaderNames()``, which returns a list of header names used as the header names in client requests.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagine getExtraHeaders is used for the runtime to quickly check if it needs to apply a more expensive call to enrich the headers. If so, add this context to the documentation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Copy link
Contributor

@tdcmeehan tdcmeehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one more thing, otherwise looks good.

@ThreadSafe
public class ClientRequestFilterManager
{
private List<ClientRequestFilterFactory> factories = new CopyOnWriteArrayList<>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make this a map.

  1. The key of the map should be the name of the factory.
  2. We should not allow duplicate names.

}

filters = factories.stream()
.map(factory -> factory.create(factory.getName()))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This construct is strange. See above comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant was, please don't pass in factory.getName() into the create method, there is no need because it can just internally call getName().

@SthuthiGhosh9400
Copy link
Contributor Author

@tdcmeehan I have made the changes. Could you please take a look?

}

filters = factories.stream()
.map(factory -> factory.create(factory.getName()))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant was, please don't pass in factory.getName() into the create method, there is no need because it can just internally call getName().

@SthuthiGhosh9400 SthuthiGhosh9400 force-pushed the request-modifier-plugin branch from a00c586 to b21ad93 Compare January 1, 2025 14:57
Copy link
Contributor

@steveburnett steveburnett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the doc! A few suggestions.

Also, to have this new doc page be visible in the Developer Guide index page, you must edit presto-docs/src/main/sphinx/develop.rst
and add
develop/client-request-filter.

@SthuthiGhosh9400 SthuthiGhosh9400 force-pushed the request-modifier-plugin branch from 02dba34 to c2bd746 Compare January 6, 2025 04:49
@SthuthiGhosh9400
Copy link
Contributor Author

@steveburnett I have incorporated the changes requested. Thanks.

steveburnett
steveburnett previously approved these changes Jan 6, 2025
Copy link
Contributor

@steveburnett steveburnett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! (docs)

Pull updated branch, new local doc build, looks good. Thanks!

@SthuthiGhosh9400
Copy link
Contributor Author

@tdcmeehan Could you take a look and approve?

tdcmeehan
tdcmeehan previously approved these changes Jan 10, 2025
@tdcmeehan
Copy link
Contributor

Please fix rebase conflicts.

@SthuthiGhosh9400
Copy link
Contributor Author

SthuthiGhosh9400 commented Jan 14, 2025

@tdcmeehan
I have resolved the conflicts during the rebase. Could you help?

@SthuthiGhosh9400
Copy link
Contributor Author

If it looks good, can we merge this PR? @tdcmeehan

@tdcmeehan
Copy link
Contributor

@SthuthiGhosh9400 the CI is read. Please investigate. If you find it's not related, rebase to rerun the test.

@SthuthiGhosh9400 SthuthiGhosh9400 force-pushed the request-modifier-plugin branch 2 times, most recently from d5df908 to 7d72a35 Compare January 16, 2025 16:49
Add ClientRequestFilter.java interface in Presto-spi module.
Improve Request Headers in the Authentication Filter Class.
@SthuthiGhosh9400
Copy link
Contributor Author

@tdcmeehan I reran the test, and it looks good now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants