-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add post "modifying json response in go"
- Loading branch information
1 parent
b4c8bcc
commit a37d007
Showing
1 changed file
with
129 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,129 @@ | ||
--- | ||
layout: post | ||
title: "Modifying `NewSingleHostReverseProxy` Response in Gin and Solving HTTP Errors" | ||
date: 2024-10-13 04:00:00 +0800 | ||
categories: code | ||
tags: golang go proxy json http gin | ||
--- | ||
|
||
## Background | ||
|
||
Setting up an HTTP proxy in Go is straightforward with the built-in [NewSingleHostReverseProxy](https://pkg.go.dev/net/http/httputil#NewSingleHostReverseProxy). When used with Gin as middleware, the code looks like this: | ||
|
||
|
||
{% highlight golang %} | ||
router := gin.New() | ||
proxy := httputil.NewSingleHostReverseProxy(lcdURL) | ||
proxyHandler := func(c *gin.Context) { | ||
proxy.ServeHTTP(c.Writer, c.Request) | ||
} | ||
router.Use(proxyHandler) | ||
{% endhighlight %} | ||
|
||
But what if we want to modify the proxied response before sending it back to the client? For simplicity, let's assume the body is in JSON format. | ||
|
||
## Built-in Functions vs. Middleware | ||
`NewSingleHostReverseProxy` doesn't directly support response modification. However, it returns a [ReverseProxy](https://pkg.go.dev/net/http/httputil#ReverseProxy) instance, allowing us to use the method `ModifyResponse`: | ||
|
||
{% highlight golang %} | ||
proxy.ModifyResponse = func(r *http.Response) error { | ||
b, err := io.ReadAll(r.Body) | ||
if err != nil { | ||
return err | ||
} | ||
defer r.Body.Close() | ||
|
||
var jsonObject map[string]interface{} | ||
err = json.Unmarshal(b, &jsonObject) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
// Modify jsonObject here | ||
|
||
newBody, err := json.Marshal(jsonObject) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
r.Body = io.NopCloser(bytes.NewReader(newBody)) | ||
r.ContentLength = int64(len(newBody)) | ||
r.Header.Set("Content-Length", strconv.Itoa(len(newBody))) | ||
return nil | ||
} | ||
{% endhighlight %} | ||
|
||
If you're running the proxy alongside other API routes, you might want a unified way to modify all responses. Writing the rewrite logic as Gin middleware can achieve this. | ||
|
||
{% highlight golang %} | ||
func filterJsonBody() gin.HandlerFunc { | ||
return func(c *gin.Context) { | ||
wb := ©Writer{ | ||
body: &bytes.Buffer{}, | ||
ResponseWriter: c.Writer, | ||
} | ||
c.Writer = wb | ||
|
||
c.Next() | ||
|
||
originBodyBytes := wb.body.Bytes() | ||
|
||
var jsonObject map[string]interface{} | ||
err := json.Unmarshal(originBodyBytes, &jsonObject) | ||
if err != nil { | ||
c.AbortWithError(http.StatusInternalServerError, err) | ||
return | ||
} | ||
|
||
// Modify jsonObject here | ||
|
||
newBody, err := json.Marshal(jsonObject) | ||
if err != nil { | ||
c.AbortWithError(http.StatusInternalServerError, err) | ||
return | ||
} | ||
|
||
wb.ResponseWriter.Write(newBody) | ||
} | ||
} | ||
|
||
type copyWriter struct { | ||
gin.ResponseWriter | ||
body *bytes.Buffer | ||
} | ||
|
||
func (cw *copyWriter) Write(b []byte) (int, error) { | ||
return cw.body.Write(b) | ||
} | ||
{% endhighlight %} | ||
|
||
Add it at the beginning of the router handler chain: | ||
|
||
{% highlight golang %} | ||
router := gin.New() | ||
router.Use(filterJsonBody()) | ||
|
||
// Other routes | ||
router.Use(proxyHandler) | ||
{% endhighlight %} | ||
|
||
## HTTP Error | ||
If you add the middleware as shown and then curl the API, you may encounter errors like: | ||
|
||
`HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)` | ||
`curl: (18) transfer closed with x bytes remaining to read` | ||
|
||
The HTTP/2 error can be cryptic, but the curl error hints at a `Content-Length` mismatch. One way to confirm the issue is to make curl show the http headers by using `curl -v` . Indeed the `Content-Length` header is set as the original body length, instead of the modified one. | ||
|
||
## Fixing the Content-Length Header in Middleware | ||
The issue is with the `Content-Length` header. A simple solution is to use `chunked` transfer encoding by overriding `WriteHeader` in `copyWriter`: | ||
|
||
{% highlight golang %} | ||
func (cw *copyWriter) WriteHeader(statusCode int) { | ||
cw.Header().Del("Content-Length") | ||
cw.Header().Set("Transfer-Encoding", "chunked") | ||
cw.ResponseWriter.WriteHeader(statusCode) | ||
} | ||
{% endhighlight %} | ||
|
||
Ideally, the `Content-Length` should match the actual response body length, but modifying it requires extra work. Some clients and proxies may not cache chunked responses properly. However, since I use Nginx in my setup, which caches small chunked responses, this tradeoff is acceptable for my use case. |