Appsmith SSL on a different port

trey · September 13, 2022, 2:41pm

New user here and love the product!
I have spent 2 days fighting this problem and am finally asking for help, so not a totally newbie question (even though this question has been asked before and answered in various forms).

I was trying to host a docker compose based install on the Digital Ocean droplet that already had ports 80 and 443 occupied.

I tried so many things which I will get into later.

For now (and for easier debugging or offering help), here is the set of simplified steps I have taken to reproduce the issue

A brand new docker compose based appsmith install with two changes in the docker-compose.yml

    ports:
      - "8080:80"
      - "8443:443"
      - "9001:9001"

Started appsmith and it is working great on port 80 over http
However, going to port 8443 over https gets a connection refused
To minimize the impact of browsers etc, I finally went old school telnet to ports 8080 and 8443 and here are the results

Port 8080
Works well and is served by the Appsmith nginx
However on Port 8443
Instantly rejected

(Sorry - new posters are only allowed one image so I had could only use one screenshot)

Questions

The question is that for a default install, is nginx listening on 443?
Does it use a self-signed cert if the APPSMITH_CUSTOM_DOMAIN has not been configured? to get started up? Or does it just reject anything on that port?

I tried to poke around the nginx conf file on github but couldnt find this clearly.

Now, I have already tried all options of ports, forwarding, custom domains etc and nothing has worked for 2 days.
So I tried to simplify this with going direct to the machine and doing everything using basic command on the localhost to eliminate firewalls, dns entries etc

Any help/pointers would be greatly appreciated.

Final Goal
My final goal is to run appsmith on a digital ocean droplet which already has another docker stack using 80/443 using caddy.
(I went down the caddy forward 80/443 route to appsmith using a domain name first, when that didnt work I resorted to isolating the problem and finally hit the connection rejected outright by appsmith on the https port)

Thanks !!

trey · September 13, 2022, 5:45pm

Quick update on further progress.
I did ssh into the appsmith installed netstat and nothing is even listening on 443 !

The nginx conf files also dont seem to have listen directive on 443 either

trey · September 13, 2022, 5:57pm

More update
After changing the Custom Domain in the UI and saving/restarting docker, I noticed that this file did change (on the docker container itself)
conf.d/nginx_app.conf.template

and the conf file now says

map $http_x_forwarded_proto $origin_scheme {
  default $http_x_forwarded_proto;
  '' $scheme;
}
# redirect log to stdout for supervisor to capture
access_log /dev/stdout;

server {
  listen 80 default_server;
  server_name my.customdomain.com;

  client_max_body_size 100m;

  gzip on;
  gzip_types *;

  server_tokens off;

It is strange there is no listen on 443 at all?
Does Appsmith NOT listen on 443?
Is there another docker container in the stack which handles that ?

pranav · September 15, 2022, 10:50am

I’ve forwarded your query to our devops team they’ll get back to you asap.

trey · September 15, 2022, 12:47pm

Thanks @pranav !
Appreciate it

sharat87 · September 17, 2022, 4:35am

Hey @trey, thanks for sharing such full details on what you’re trying to achieve, what you tried to get there, and what failed. Such details makes it so much easier to provide exactly the help you need. Appreciate the effort you’ve put in this.

From what I’ve read, I think you are missing a few pieces of information that would’ve helped figure this out for yourself.

Appsmith dynamically generates the NGINX configuration at container startup. At container startup, if Appsmith figures that HTTPS is possible, it’ll generate the config accordingly. In your case, that wouldn’t be happening, and so the config is generated only for HTTP, which means, no listening on port 443.

So the next question is, why is Appsmith unable to setup HTTPS? This follows a few simple rules:

If APPSMITH_CUSTOM_DOMAIN is set, then on startup, Appsmith tries to provision a SSL certificate for that domain. If this fails, it’ll decide that HTTPS is not happening, and just start with HTTP.
This provisioning works with certbot, very similar to how Caddy auto-provisions SSL certificates. But the nuance here is that, these methods only work if port 80 is available to bind to. If port 80 is occupied for something else, this method of provisioning a certificate will not work. I can go into why this is the case, if you’re interested, but it’s a tangent at this point.

Now there’s two things you can/may do to move ahead.

If you can afford it, make port 80 and 443 available for a few minutes, restart Appsmith container, let it provision its certificates and start with HTTPS. Then you can close ports 80 and 443 again.
- Hiccups with this: One, you’ll need to change the port bindings in docker-compose.yml to the defaults. You can change them back after cert provisioning is done. Two, the provisioned cert will be valid for ~3 months. At that time, you’ll have to do this whole dance again.
Provision your SSL certificate elsewhere, through any other methods, and give it to Appsmith. Once you have the cert copied to your server, use instructions from https://docs.appsmith.com/getting-started/setup/instance-configuration/custom-domain#custom-ssl-certificate to give it to Appsmith.
- From what I gather, this is likely the most comfortable option for you. Since you have another service running on port 80 and 443, I’m willing to bet that you already have a cert for your domain. Just share it with Appsmith and it’ll make the best use of it.

Let me know if you have further troubles or if you want me to elaborate on something from above.

Thank you for writing to us!

trey · September 17, 2022, 3:02pm

@sharat87 - Thanks a ton for the detailed explanation.
This all makes sense and it was what I was suspecting was the case. All valid - cant server SSL unless it has a use.

Now let me get back to my original problem that led me here.
As I mentioned I have a Caddy server on 80 and 443 and now I just want to use that as a reverse proxy and also terminate the SSL for Appsmith (thats the default config for Caddy).

Internet:80/443 → Caddy 80/443 (HTTP for Certgen and redirect to HTTPS) → Appsmith (0.0.0.0:8080 HTTP)

I am using n8n which a great backend complement to appsmiths front end - very amazing combo

Obviously to make this work, i will use two domain names pointed to the same ip

I kept the caddy config simple

n8n.example.com {
    reverse_proxy n8n:5678
}

appsmith.example.com:443 {
   reverse_proxy 0.0.0.0:8080 
}

I tried various options like

https://appsmith.example.com {
   reverse_proxy 0.0.0.0:8080 
}

And added various X-Forwarded directives in the header.
I tried various internal IPs and docker network - ie appsmith_default for the redirection or just :8080 etc
NOTHING mattered.

I kept getting Error 502 from Caddy.

I tested to see if Caddy was working and tried these directives
respond "Hello, world!"
That worked.
Then I redirected reverse_proxy to google.com
That also worked.
So Caddy is working fine - it has the SSL Cert for appsmith.example.com

BUT when I try to make Caddy reverse_proxy plain HTTP based appsmith - it does NOT work and keeps sending 502 - upstream error

For reference here is the appsmith docker ports

    ports:
      - "8080:80"
      - "8443:443"
      - "9001:9001"

I checked the ports are listening to the IPs using netstat

tcp        0      0 0.0.0.0:9001            0.0.0.0:*               LISTEN      395623/docker-proxy
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      983834/docker-proxy
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      395691/docker-proxy
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      983817/docker-proxy
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      395671/docker-proxy
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      140393/docker-proxy
tcp6       0      0 :::9001                 :::*                    LISTEN      395631/docker-proxy
tcp6       0      0 :::80                   :::*                    LISTEN      983839/docker-proxy
tcp6       0      0 :::8080                 :::*                    LISTEN      395698/docker-proxy
tcp6       0      0 :::443                  :::*                    LISTEN      983822/docker-proxy
tcp6       0      0 :::8443                 :::*                    LISTEN      395678/docker-proxy
tcp6       0      0 :::8000                 :::*                    LISTEN      140400/docker-proxy

Simple question - how do I make Caddy terminate SSL and just serve Appsmith reverse_proxy non SSL
Seems so simple it should work - not sure whats going on

Thanks

trey · September 17, 2022, 3:52pm

@sharat87 - I used the SSL link for self-provided ssl certs and that worked on port 8443.
In fact I just copied the caddy ssl certs that it had acquired for appsmith.example.com over to appsmith/stacks/ssl

Still cant get the Caddy reverse-proxy to appsmith over HTTP or HTTPS - no protocol works

trey · September 17, 2022, 6:05pm

Ok - got it to work.
The docker container for caddy did not know how to reach the appsmith container - duh !
Here is the caddyfile that worked

https://appsmith.example.com {
        reverse_proxy http://10.128.22.55:8080 {
    }
}

Here the 10.128.22.55 address is the address of the host machine thats listening on 8080 using the appsmith docker_compose file

Special attention to incoming directive https:// reverse proxying to http:// (no S) - that took me a while to get working

ports:
      - "8080:80"
      - "8443:443"

I am sure there are more elegant solutions like custom bridge networks, adding the whole appsmith and n8n to a single docker_compose and hence the networking etc.
But this barebones approach works for now.

IMPORTANT - I had to remove the APPSMITH_CUSTOM_DOMAIN configuration in Appsmith (Through the UI - Settings - Advanced) so that as @sharat87 said - Appsmith recognizes there is no SSL so it stops nginx from redirecting from 80 to 443 which is not what I need for Caddy to reverseproxy

Thanks for the help

sharat87 · September 19, 2022, 9:46am

Hey @trey, glad you have this working, but… somewhere I think you may be complicating things for yourself. If I remember correctly, a config with just

n8n.mydomain.com {
  reverse_proxy n8n:5678
}

appsmith.mydomain.com {
  reverse_proxy appsmith
}

Or something like that should work just fine. Just ensure that in your docker-compose.yml file, the key name of appsmith service is appsmith.

Also ensure APPSMITH_CUSTOM_DOMAIN is not set in stacks/configuration/docker.env, which actually the default, you don’t need it for your setup.

As for the ports section in docker-compose.yml under appsmith, you don’t need it all. Instead, replace it with this:

expose:
  - "80"

I say this might be simpler for you since I see you have (what I believe) a hard-coded IP belonging to Docker’s default bridge network. With the method I detailed above, you shouldn’t need that.

trey · September 19, 2022, 12:06pm

This is what I tried before and it didnt work.
I even tried to ping “appsmith” from the Caddy docker comtainer - it doesnt resolve.
The reason I believe is that these are two separate docker stacks and cannot resolve names between each other.

I unfortunately dont know docker enough to make the name resolution work or make the appsmith and n8n stacks talk to each other, but I will keep trying till I figure it out.

Thanks for all your help.

trey · September 19, 2022, 12:29pm

The final piece
To avoid hardcoding, here is what I did
I passed an extra_hosts name to caddy in the docker_compose.yml for n8n (Windows seems to do this by default but Ubuntu does not)

    extra_hosts:
      - "host.docker.internal:host-gateway"

Then Caddy can resolve the name host.docker.internal to the Docker host and changed the Caddy file to

https://appsmith.example.com {
    reverse_proxy http://host.docker.internal:8080
}

No hardcoding and it all works !

sharat87 · September 20, 2022, 1:36pm

Okay, I think you have two separate docker-compose.yml files here, or more, right? What I suggested would only work if you have all three, Caddy, n8n, and Appsmith, in the same docker-compose.yml. Should’ve specified that, sorry.

Glad you don’t have a hard-code anymore though.