Early Rejection of Large Requests Based on Content-Length Header #6032

yzfeng2020 · 2024-12-12T05:07:11Z

Motivation:

Currently, large requests are rejected only after the transferred bytes exceed the configured limit. This behavior is sub-optimal for requests that include a valid Content-Length header indicating the request is already too large. By rejecting such requests earlier—when the header is read—we can improve resource utilization and reduce unnecessary processing.

Modifications:

Added a field to ContentTooLargeException to indicate when the exception is raised during header processing.
Implemented content-length-based early rejection in Http1ObjectDecoder and Http2ObjectDecoder.
Result:

Result:

Closes Early rejection for requests exceeding configured maxRequestLength #5880
Requests with a Content-Length header exceeding the allowed limit can now be rejected early in the request flow, reducing wasted resources and improving efficiency.

github-actions · 2024-12-12T05:28:16Z

🔍 Build Scan® (commit: `3636dff`)

Job name	Status	Build Scan®
build-ubicloud-standard-8-jdk-8	✅	https://ge.armeria.dev/s/pl6iuoy2rdclu
build-ubicloud-standard-8-jdk-21-snapshot-blockhound	✅	https://ge.armeria.dev/s/rnfdrrqdtpzc6
build-ubicloud-standard-8-jdk-17-min-java-17-coverage	✅	https://ge.armeria.dev/s/2zfrlryxrrjn4
build-ubicloud-standard-8-jdk-17-min-java-11	✅	https://ge.armeria.dev/s/swr2cth4lskdg
build-ubicloud-standard-8-jdk-17-leak	✅	https://ge.armeria.dev/s/3jvp3y4ffgapi
build-ubicloud-standard-8-jdk-11	✅	https://ge.armeria.dev/s/jpiig5yllvtnc
build-macos-latest-jdk-21	✅	https://ge.armeria.dev/s/pqjmcco2kbakm

yzfeng2020 · 2024-12-12T08:25:02Z

core/src/test/java/com/linecorp/armeria/server/HttpServerTest.java

@@ -600,8 +600,8 @@ void testTimeoutAfterPartialContentWithPooling(WebClient client) throws Exceptio
    void testTooLargeContentToNonExistentService(WebClient client) {
        final byte[] content = new byte[(int) MAX_CONTENT_LENGTH + 1];
        final AggregatedHttpResponse res = client.post("/non-existent", content).aggregate().join();
-        assertThat(res.status()).isSameAs(HttpStatus.NOT_FOUND);
-        assertThat(res.contentUtf8()).startsWith("Status: 404\n");
+        assertThat(res.status()).isSameAs(HttpStatus.REQUEST_ENTITY_TOO_LARGE);


i had to change this test as it's rejected earlier now, is there a way to fix it

To simulate a client request with no content-length header, we can probably make a transfer-encoding request.

e.g.

val req = HttpRequest.streaming(POST, "/non-existent); client.execute(req)... req.write(content) req.close();

make sense, updated!

core/src/main/java/com/linecorp/armeria/common/ContentTooLargeException.java

yzfeng2020 · 2024-12-12T08:26:05Z

core/src/main/java/com/linecorp/armeria/common/DefaultFlagsProvider.java

@@ -472,6 +472,11 @@ public Boolean allowSemicolonInPathComponent() {
        return false;
    }

+    @Override
+    public Boolean allowLargeRequestEarlyRejection() {


I tend to guard this feature under a flag..

core/src/main/java/com/linecorp/armeria/server/Http1RequestDecoder.java

jrhee17

Looks good overall, left some minor comments

core/src/main/java/com/linecorp/armeria/common/ContentTooLargeExceptionBuilder.java

core/src/main/java/com/linecorp/armeria/server/Http1RequestDecoder.java

core/src/main/java/com/linecorp/armeria/server/Http2RequestDecoder.java

core/src/main/java/com/linecorp/armeria/common/ContentTooLargeExceptionBuilder.java

jrhee17

👍 👍 👍

yzfeng2020 · 2025-01-01T04:01:19Z

@ikhoon @minwoox happy new year! I hope you can review this PR when you have time, thanks

minwoox

@yzfeng2020 Happy new year! 😆

core/src/main/java/com/linecorp/armeria/server/Http1RequestDecoder.java

yzfeng2020 · 2025-01-06T08:32:43Z

hey @ikhoon gentle ping if you can take a look! :)

core/src/main/java/com/linecorp/armeria/common/ContentTooLargeExceptionBuilder.java

ikhoon · 2025-01-07T08:03:22Z

core/src/main/java/com/linecorp/armeria/server/Http1RequestDecoder.java

+                    if (maxRequestLength > 0 && contentLength > maxRequestLength) {
+                        abortLargeRequest(ctx, req, id, endOfStream, keepAliveHandler, true);
+                    }


I am unsure if it is necessary to pass the content-length exceeding request to the service chains. Should we fail() the request instead? What do you think @minwoox?

IIRC, it was implemented this way to meet a demand for logging content-length-exceeding requests via LoggingService.

This feature could still be useful for identifying such requests on the server side, provided it doesn't cause a performance issue. 🤔

I wasn't worried about performance, but I was considering the race condition between the response sent by the user and the aborted response.

That case would be rare, so I think it can be ignored.

Yeah, let's revisit this if we encounter any issue with this. 😉

…ExceptionBuilder.java

minwoox · 2025-01-09T00:54:49Z

@yzfeng2020 Thanks! 👍 👍 👍

Motivation: Currently, large requests are rejected only after the transferred bytes exceed the configured limit. This behavior is sub-optimal for requests that include a valid Content-Length header indicating the request is already too large. By rejecting such requests earlier—when the header is read—we can improve resource utilization and reduce unnecessary processing. Modifications: - Added a field to ContentTooLargeException to indicate when the exception is raised during header processing. - Implemented content-length-based early rejection in Http1ObjectDecoder and Http2ObjectDecoder. Result: Result: - Closes #5880 - Requests with a Content-Length header exceeding the allowed limit can now be rejected early in the request flow, reducing wasted resources and improving efficiency.

yzfeng2020 force-pushed the early-reject-large-req branch from fd7fe9e to 0a25899 Compare December 12, 2024 08:20

yzfeng2020 commented Dec 12, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/common/ContentTooLargeException.java Outdated Show resolved Hide resolved

yzfeng2020 commented Dec 12, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/Http1RequestDecoder.java Show resolved Hide resolved

yzfeng2020 commented Dec 12, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/Http1RequestDecoder.java Outdated Show resolved Hide resolved

yzfeng2020 marked this pull request as ready for review December 13, 2024 08:55

yzfeng2020 requested review from ikhoon, jrhee17, minwoox and trustin as code owners December 13, 2024 08:55

jrhee17 reviewed Dec 18, 2024

View reviewed changes

yzfeng2020 force-pushed the early-reject-large-req branch from 36bb54d to c114764 Compare December 20, 2024 06:27

yzfeng2020 changed the title ~~early reject~~ Early Rejection of Large Requests Based on Content-Length Header Dec 20, 2024

yzfeng2020 commented Dec 20, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/common/ContentTooLargeExceptionBuilder.java Show resolved Hide resolved

yzfeng2020 commented Dec 20, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/common/ContentTooLargeExceptionBuilder.java Outdated Show resolved Hide resolved

feat: early reject large request

990f4cb

yzfeng2020 force-pushed the early-reject-large-req branch from 06f9c52 to 990f4cb Compare December 20, 2024 09:42

yzfeng2020 requested a review from jrhee17 December 20, 2024 09:43

jrhee17 approved these changes Dec 30, 2024

View reviewed changes

minwoox added the defect label Jan 2, 2025

minwoox added this to the 1.32.0 milestone Jan 2, 2025

minwoox approved these changes Jan 2, 2025

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/Http1RequestDecoder.java Outdated Show resolved Hide resolved

lint

7167784

ikhoon reviewed Jan 7, 2025

View reviewed changes

ikhoon added 2 commits January 8, 2025 10:09

Update core/src/main/java/com/linecorp/armeria/common/ContentTooLarge…

3e9a3b5

…ExceptionBuilder.java

Update core/src/main/java/com/linecorp/armeria/common/ContentTooLarge…

3636dff

…ExceptionBuilder.java

ikhoon approved these changes Jan 8, 2025

View reviewed changes

minwoox merged commit 1157370 into line:main Jan 9, 2025
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Early Rejection of Large Requests Based on Content-Length Header #6032

Early Rejection of Large Requests Based on Content-Length Header #6032

yzfeng2020 commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

yzfeng2020 Dec 12, 2024 •

edited

Loading

jrhee17 Dec 18, 2024

yzfeng2020 Dec 20, 2024

yzfeng2020 Dec 12, 2024

jrhee17 left a comment

jrhee17 left a comment

yzfeng2020 commented Jan 1, 2025

minwoox left a comment

yzfeng2020 commented Jan 6, 2025

ikhoon Jan 7, 2025

minwoox Jan 7, 2025

ikhoon Jan 7, 2025 •

edited

Loading

minwoox Jan 8, 2025

minwoox commented Jan 9, 2025

Early Rejection of Large Requests Based on Content-Length Header #6032

Early Rejection of Large Requests Based on Content-Length Header #6032

Conversation

yzfeng2020 commented Dec 12, 2024 • edited Loading

github-actions bot commented Dec 12, 2024 • edited Loading

🔍 Build Scan® (commit: 3636dff)

yzfeng2020 Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

jrhee17 Dec 18, 2024

Choose a reason for hiding this comment

yzfeng2020 Dec 20, 2024

Choose a reason for hiding this comment

yzfeng2020 Dec 12, 2024

Choose a reason for hiding this comment

jrhee17 left a comment

Choose a reason for hiding this comment

jrhee17 left a comment

Choose a reason for hiding this comment

yzfeng2020 commented Jan 1, 2025

minwoox left a comment

Choose a reason for hiding this comment

yzfeng2020 commented Jan 6, 2025

ikhoon Jan 7, 2025

Choose a reason for hiding this comment

minwoox Jan 7, 2025

Choose a reason for hiding this comment

ikhoon Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

minwoox Jan 8, 2025

Choose a reason for hiding this comment

minwoox commented Jan 9, 2025

yzfeng2020 commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

🔍 Build Scan® (commit: `3636dff`)

yzfeng2020 Dec 12, 2024 •

edited

Loading

ikhoon Jan 7, 2025 •

edited

Loading