Unable to access params array when running behind nginx reverse proxy with ssl

[ksteimel]

Describe the bug
elements in@params are not accessible after form submission for genie app running behind nginx reverse proxy with ssl enabled. Using ssh port forwarding directly to app works flawlessly and elements of @params are accessible.

To Reproduce

• Start app with form where an element of @params is accessed by function receiving post action (simply printing will work)
• Run app behind nginx reverse proxy with ssl enabled
• Submit form
• KeyError is raised for the element in @params you were trying to access

Expected behavior
The element in @params should be accessible as it is when not behind the nginx reverse proxy.

Additional context

julia> versioninfo()

Julia Version 1.1.1
Commit 55e36cc308 (2019-05-16 04:10 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, westmere)

pkg> st

    Status `~/ParsingGenie/Project.toml`
  [1520ce14] + AbstractTrees v0.2.1
  [62bf7c00]   CFG v0.1.0 #master (https://git.ksteimel.duckdns.org/ksteimel/CFG.jl.git)
  [c43c736e]   Genie v0.9.2 #master (https://github.com/essenciary/Genie.jl)
  [682c06a0]   JSON v0.20.0
  [39ec1447]   Millboard v0.2.0
  [295af30f]   Revise v2.1.5

Nginx reverse proxy config

Note that app is configured to start on port 2112

server {
        listen 443 ssl http2;

        server_name parser.ksteimel.duckdns.org;

        location / {
                proxy_pass      http://192.168.1.104:2112;
                proxy_redirect  off;
                proxy_set_header Host $host;
                proxy_buffering     on;
                proxy_set_header    X-Real-IP $remote_addr;
                proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header    X-Forwarded-Proto $scheme;
                proxy_set_header    X-Forwarded-Host $server_name;
        }

    client_max_body_size 1G;


    ssl_certificate /etc/letsencrypt/live/parser.ksteimel.duckdns.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/parser.ksteimel.duckdns.org/privkey.pem; # managed by Certbot
}

The app I'm running can be found here

[essenciary]

@ksteimel Thanks! Do you have a Dockerfile or something so I can replicate the environment (I'm on mac).

[ksteimel]

I don't have a dockerfile preapred :/ I was using lxc containers to isolate it. Do you need a container for both nginx and the app or for just the app?

[essenciary]

OK, don't worry - I have it on my TODO list to make a Dockerfile for Genie apps (Julia, Nginx proxy, SQLite, Postgres, Redis, Memcache, etc). This looks like a good time to do it - but it will take a bit longer to set it up.

[ksteimel]

Sorry! Let me know if I can help. I'm decently competent with Docker.

[essenciary]

@ksteimel If you have the time and are happy to contribute a docker image for Genie users, that would be amazing. It's something we needed/wanted for a long time and we'd push it the way to install the full stack.

In terms of spec, must haves are:

• Julia 1.1.x
• SQLite [for dev]
• Postgres [for prod]
• Nginx - on port 80 configured as a reverse proxy for the Genie apps running on 8080 and SSL (Let's encrypt)
• Supervisord to start Genie apps as services (and autostart on crash)
• Nodejs [for asset management]
• based off Ubuntu?

Really good to have:

• some control panel to manage services (maybe https://cockpit-project.org or similar?)

Nice to have:

• Memcache [for caching and/or session storage]
• Redis [for caching and/or session storage]
• MySQL [for dev / prod]

Anything else? Utilities for logging error, other interesting monitoring tools for when running in production, etc?

What do you think? Maybe we can start with an image for web dev and modify it for Julia?

[ksteimel]

#response_container_BBPPID{font-family: initial; font-size:initial; color: initial;} Hmm do you think it makes more sense to put all of that in a single docker or use a docker-compose? Thank you,  Kenneth Steimel  Associate Instructor  Indiana University From: [email protected]: June 20, 2019 10:24 AMTo: [email protected]: [email protected]: [email protected]; [email protected]: Re: [GenieFramework/Genie.jl] Unable to access params array when running behind nginx reverse proxy with ssl (#145) @ksteimel If you have the time and are happy to contribute a docker image for Genie users, that would be amazing. It's something we needed/wanted for a long time and we'd push it the way to install the full stack. In terms of spec, must haves are: Julia 1.1.xSQLite [for dev]Postgres [for prod]Nginx - on port 80 configured as a reverse proxy for the Genie apps running on 8080 and SSL (Let's encrypt)Supervisord to start Genie apps as services (and autostart on crash)Nodejs [for asset management]based off Ubuntu? Really good to have: some control panel to manage services (maybe https://cockpit-project.org or similar?) Nice to have: Memcache [for caching and/or session storage]Redis [for caching and/or session storage]MySQL [for dev / prod] Anything else? Utilities for logging error, other interesting monitoring tools for when running in production, etc? What do you think? Maybe we can start with an image for web dev and modify it for Julia? —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[essenciary]

@ksteimel Sorry, didn't have time to look into this till now. I've never used docker-compose but I checked it out now and it looks great.

[Ankur-deDev]

Hi Genie,
Bumped into the same issue, I noticed it worked when removing http2 from the nginx configuration.
I did not investigate further, but this issue might be related.

[essenciary]

@Ankur-deDev Thanks - can you please show a dump of the headers? Like the full request maybe? I presume proxying messes up with the headers...

dump(@params(:REQUEST))

And maybe the same for the full @params object?

Thanks

[ksteimel]

#response_container_BBPPID{font-family: initial; font-size:initial; color: initial;} Oh yeah I forgot I found a work around for this. The values are in rawpayload if running behind an ssl enable nginx reverse proxy. You can then run `URIParser.query_params(rawpayload())` To get the resulting dict (with the exception that the keys are now strings).  From: [email protected]: November 14, 2019 3:33 AMTo: [email protected]: [email protected]: [email protected]; [email protected]: [External] Re: [GenieFramework/Genie.jl] Unable to access params array when running behind nginx reverse proxy with ssl (#145) This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.@Ankur-deDev Thanks - can you please show a dump of the headers? Like the full request maybe? I presume proxying messes up with the headers... dump(@params(:REQUEST)) And maybe the same for the full @params object? Thanks —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[essenciary]

@ksteimel Thanks for letting me know! If you can pass me the rawpayload maybe I can make a generic fix.

[ksteimel]

This is what I had in my logs for the output of rawpayload()

rules=S+-%3E+NP+VP+%0D%0ANP+-%3E+D+N+%0D%0AVP+-%3E+V+%0D%0AN+%3A+dog%0D%0AD+%3A+the%0D%0AV%3A+ran%0D%0A%23This+is+a+comment&sentences=the+dog+ran&action_type=parse&generate_n=1

[essenciary]

Hmm... I wonder if it's a content type issue. Do you explicitly set the content type of the form? Ie application/x-www-form-urlencoded or multipart/form-data?

[ksteimel]

#response_container_BBPPID{font-family: initial; font-size:initial; color: initial;} I do not. Haha. Do you want me to try that and see what happens? I can once I get off work.  From: [email protected]: November 14, 2019 10:20 AMTo: [email protected]: [email protected]: [email protected]; [email protected]: [External] Re: [GenieFramework/Genie.jl] Unable to access params array when running behind nginx reverse proxy with ssl (#145) This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.Hmm... I wonder if it's a content type issue. Do you explicitly set the content type of the form? Ie application/x-www-form-urlencoded or multipart/form-data? —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[essenciary]

Yes please, if you can :)

[ksteimel]

Sorry for the delay, setting enctype="application/x-www-form-urlencoded" has no effect. I still have to parse the return from raw_payload()

[essenciary]

Bummer - probably easy to solve if URIParser.query_params knows how to handle it. The difficult part is reproducing the bug as I have to set up the whole stack.

[essenciary]

Solved!