Skip to content

Commit

Permalink
Add post "modifying json response in go"
Browse files Browse the repository at this point in the history
  • Loading branch information
williamchong committed Oct 13, 2024
1 parent b4c8bcc commit a37d007
Showing 1 changed file with 129 additions and 0 deletions.
129 changes: 129 additions & 0 deletions _posts/2024-10-13-modifying-json-response-in-go.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
---
layout: post
title: "Modifying `NewSingleHostReverseProxy` Response in Gin and Solving HTTP Errors"
date: 2024-10-13 04:00:00 +0800
categories: code
tags: golang go proxy json http gin
---

## Background

Setting up an HTTP proxy in Go is straightforward with the built-in [NewSingleHostReverseProxy](https://pkg.go.dev/net/http/httputil#NewSingleHostReverseProxy). When used with Gin as middleware, the code looks like this:


{% highlight golang %}
router := gin.New()
proxy := httputil.NewSingleHostReverseProxy(lcdURL)
proxyHandler := func(c *gin.Context) {
proxy.ServeHTTP(c.Writer, c.Request)
}
router.Use(proxyHandler)
{% endhighlight %}

But what if we want to modify the proxied response before sending it back to the client? For simplicity, let's assume the body is in JSON format.

## Built-in Functions vs. Middleware
`NewSingleHostReverseProxy` doesn't directly support response modification. However, it returns a [ReverseProxy](https://pkg.go.dev/net/http/httputil#ReverseProxy) instance, allowing us to use the method `ModifyResponse`:

{% highlight golang %}
proxy.ModifyResponse = func(r *http.Response) error {
b, err := io.ReadAll(r.Body)
if err != nil {
return err
}
defer r.Body.Close()

var jsonObject map[string]interface{}
err = json.Unmarshal(b, &jsonObject)
if err != nil {
return err
}

// Modify jsonObject here

newBody, err := json.Marshal(jsonObject)
if err != nil {
return err
}

r.Body = io.NopCloser(bytes.NewReader(newBody))
r.ContentLength = int64(len(newBody))
r.Header.Set("Content-Length", strconv.Itoa(len(newBody)))
return nil
}
{% endhighlight %}

If you're running the proxy alongside other API routes, you might want a unified way to modify all responses. Writing the rewrite logic as Gin middleware can achieve this.

{% highlight golang %}
func filterJsonBody() gin.HandlerFunc {
return func(c *gin.Context) {
wb := &copyWriter{
body: &bytes.Buffer{},
ResponseWriter: c.Writer,
}
c.Writer = wb

c.Next()

originBodyBytes := wb.body.Bytes()

var jsonObject map[string]interface{}
err := json.Unmarshal(originBodyBytes, &jsonObject)
if err != nil {
c.AbortWithError(http.StatusInternalServerError, err)
return
}

// Modify jsonObject here

newBody, err := json.Marshal(jsonObject)
if err != nil {
c.AbortWithError(http.StatusInternalServerError, err)
return
}

wb.ResponseWriter.Write(newBody)
}
}

type copyWriter struct {
gin.ResponseWriter
body *bytes.Buffer
}

func (cw *copyWriter) Write(b []byte) (int, error) {
return cw.body.Write(b)
}
{% endhighlight %}

Add it at the beginning of the router handler chain:

{% highlight golang %}
router := gin.New()
router.Use(filterJsonBody())

// Other routes
router.Use(proxyHandler)
{% endhighlight %}

## HTTP Error
If you add the middleware as shown and then curl the API, you may encounter errors like:

`HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)`
`curl: (18) transfer closed with x bytes remaining to read`

The HTTP/2 error can be cryptic, but the curl error hints at a `Content-Length` mismatch. One way to confirm the issue is to make curl show the http headers by using `curl -v` . Indeed the `Content-Length` header is set as the original body length, instead of the modified one.

## Fixing the Content-Length Header in Middleware
The issue is with the `Content-Length` header. A simple solution is to use `chunked` transfer encoding by overriding `WriteHeader` in `copyWriter`:

{% highlight golang %}
func (cw *copyWriter) WriteHeader(statusCode int) {
cw.Header().Del("Content-Length")
cw.Header().Set("Transfer-Encoding", "chunked")
cw.ResponseWriter.WriteHeader(statusCode)
}
{% endhighlight %}

Ideally, the `Content-Length` should match the actual response body length, but modifying it requires extra work. Some clients and proxies may not cache chunked responses properly. However, since I use Nginx in my setup, which caches small chunked responses, this tradeoff is acceptable for my use case.

0 comments on commit a37d007

Please sign in to comment.