Appsmith does not connect to external MongoDB, exited with code 132

arminus · February 25, 2023, 2:47pm

As a consequence of the rejected bug report here and the originating problem of not being able to run Appsmith > 1.8.5 on a Pi4 anymore, I’m trying to solve by running a custom built Mongo5 instance and connect to it from Appsmith latest.

I’m basically setting APPSMITH_MONGODB_URI as documented here, but all I’m getting is this:

appsmith    | ++ mongo --host mongodb+srv://mongodb/appsmith --quiet --eval 'rs.status().ok'
appsmith    | + mongo_state=
appsmith exited with code 132

The setup (i.e. Dockerfile for the custom MongoDB 5 and the compose for the stack) is here: GitHub - arminus/appsmith-docker - together with a full log in Appsmith does not connect to external mongodb instance · Issue #1 · arminus/appsmith-docker · GitHub

Am I missing something in regards to maybe having to further initialize the empty MongoDB? What else could be causing the 132 exit? Could this be a crashing 5.x mongo client binary in the Appsmith image?

Based on the suggestion in [Bug]: Cannot upgrade beyond 1.8.5 on ARM due to MongoDB 5 · Issue #20818 · appsmithorg/appsmith · GitHub “please use an external MongoDB”, my assumption was that it’s not the Appsmith image/code itself which “has a problem” on Pi4… (and with my setup, I am basically just trying to use an obviously functioning external MongoDB → I can connect to it from the outside with Compass)

arminus · February 26, 2023, 12:55pm

So I debugged this further. Couple of observations:

apparently, appsmithctl check-replica-set uses an “original” mongo5 binary which core dumps on the Pi4/arm64
so I copied the working/unofficial mongo5 binary into my own appsmith image, still failing:

appsmith    | ++ mongo --host mongodb://mongodb/appsmith --quiet --eval 'rs.status().ok'
appsmith    | + mongo_state='Error: couldn'\''t connect to server mongodb:27017, connection attempt failed: SocketException: Error connecting to mongodb:27017 (172.22.0.2:27017) :: caused by :: Connection refused :

I do compose start my mongo5 together with appsmith, appsmith has a depend_on condition on mongo, but I think mongo isn’t fully up at that point, because:

Next I copied a modified entrypoint.sh into my appsmith image which skips the appsmithctl check-replica-set step and all of a sudden, everything works
shelling into that appsmith image and manually running the check from above works (given a couple of seconds wait until everything is up)

root@9fa1c00190a9:/opt/appsmith# mongo --host $APPSMITH_MONGODB_URI --eval 'rs.status().ok'
MongoDB shell version v5.0.5
connecting to: mongodb://mongodb:27017/appsmith?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("4afbc29c-6ea4-4f89-aec9-37aabb6a9ffe") }
MongoDB server version: 5.0.5
1

So, I have a kind of solution by basically patching the official appsmith-ce image but at least the appsmithctl check-replica-set being called to early in a situation where both mongo and appsmith are started with the same stack seems to be a general problem.

Also, are the mongo binaries in that image used for anything else if in fact an external mongo is in use? (that might cause issues downstream which I’m not aware of yet)

Amelia · February 27, 2023, 5:04am

Hello Arminus,

The way to make the latest Appsmith version work with an external MongoDB 5 is to first connect to an external MongoDB while still on Appsmith v1.8.15. After that, you’d need to update your instance to Appsmith v1.9.2, which is our checkpoint version, as per the steps from this guide. Then you need to update to the latest release.
Can you please confirm if these are the steps you have taken?

arminus · February 27, 2023, 7:15am

No I didn’t. This is a from-scratch setup test.

Amelia · February 27, 2023, 8:48am

Ok, got it. So this is a new Appsmith installation with a custom MongoDB. Please send us the server logs to help us investigate the issue. You can grab the logs by running docker logs -f appsmith. Or you could follow the steps from this guide on how to get the logs

arminus · February 27, 2023, 9:38am

I cannot upload a .log file here, here it is linked to my github ticket: https://github.com/arminus/appsmith-docker/files/10831373/startup-fail.log

But that really doesn’t include more in addition to what I had outlined above…

Amelia · February 28, 2023, 4:55am

Thank you, Arminus. Our team has investigated and found that it’s an issue on our side. This will be fixed on priority - our team has already created a PR.

arminus · March 3, 2023, 2:42pm

I’m confused. Using mongosh instead of mongo in init_replica_set is not the point here…

To rephrase what I had mentioned above:

out-of-the box, appsmith doesn’t start on a Pi4/arm with an external Mongo5 DB because internally, it uses a core dumping mongo5 binary (problem 1) in appsmithctl check-replica-set and it’s called to early (problem 2).

Looking at this change, I see nothing which would fix that problem.

In other words, it will still fail on a Pi4/arm even if it uses an external Mongo5.

So the questions remain: Can you fix the timing problem? And can you do the appsmithctl check-replica-set without relying on a binary which doesn’t work on Pi47arm - or alternatively let the user disable that via an env var?

As I said, I do have a workround for all this in GitHub - arminus/appsmith-docker but this could be superfluous if the above issues would be fixed on you side…

(I’m just trying to make appsmith available for folks with are also deploying on a Pi4/arm)

sharat87 · March 6, 2023, 1:08pm

@arminus, thanks for the details. The linked change is actually the last place the mongo binary is used, and is where the container fails to start. So we’re fixing that.

But,

[…] uses a core dumping mongo5 binary (problem 1) in appsmithctl check-replica-set […]

Can you elaborate on where the check-replica-set command is using the mongo binary? It uses the NodeJS MongoDB SDK to connect and check, not the mongo binary. So it shouldn’t be affected by whether that binary can work on your system or not.

arminus · March 6, 2023, 5:51pm

You are right, based on the log message when starting the container:

appsmith    | ++ mongo --host mongodb://mongodb/appsmith --quiet --eval 'rs.status().ok'

I assumed it was trying to use the mongo binary… Maybe this was the original entrypoint.sh file form the original image?

The entrypoint.sh I’m using for further tests is the latest from git…

So I checked again, now it turns out that appsmithctl check-replica-set tries to connect to 127.0.0.1 instead of the mongodb container: This can be observed in the startup log, as well as when I shell directly into the running image (after I’ve commented out the appsmithctl check-replica-set so that it doesn’t get shut down)

root@4f45d009f807:/opt/appsmith# echo $APPSMITH_MONGODB_URI 
mongodb://mongodb/appsmith
root@4f45d009f807:/opt/appsmith# appsmithctl check-replica-set
Error trying to check replicaset MongoServerSelectionError: connect ECONNREFUSED 127.0.0.1:27017
    at Timeout._onTimeout (/opt/appsmith/utils/node_modules/mongodb/lib/sdam/topology.js:312:38)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7) {
  reason: TopologyDescription {
    type: 'ReplicaSetNoPrimary',
    servers: Map(1) { 'localhost:27017' => [ServerDescription] },
    stale: false,
    compatible: true,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    setName: 'rs0',
    maxSetVersion: 1,
    maxElectionId: new ObjectId("7fffffff0000000000000011"),
    commonWireVersion: 13,
    logicalSessionTimeoutMinutes: undefined
  }
}

which is really strange because /usr/lib/node_modules/appsmith_utils/bin/check_replica_set.js does initialize the client with
const client = new MongoClient(process.env.APPSMITH_MONGODB_URI, …)

?!

sharat87 · March 7, 2023, 1:05am

The entrypoint.sh I’m using for further tests is the latest from git…

By latest from git, you mean appsmith/entrypoint.sh at release · appsmithorg/appsmith · GitHub, right? But it doesn’t have that mongo --host ... command. Can you may be fetch it again and try your tests? ~~Although I think the one included with the appsmith/appsmith-ce:latest image should work fine for you.~~ Edit: The PR is still not merged I see. I’ll get to it soon.

turns out that appsmithctl check-replica-set tries to connect to 127.0.0.1

This is primarily because of two things.

The docker.env file is not given in docker-compose.yml, but is source-ed in the entrypoint script. This means the env variables aren’t available when we open a shell into the container. We have to source /appsmith-stacks/configuration/docker.env explicitly.
For the appsmithctl command to be able to access this env variable, it needs to be exported. Just source-ing the docker.env file, unfortunately, doesn’t export it. You can do export APPSMITH_MONGODB_URI after the source command, and then try appsmithctl check-replica-set, or use the following command to have everything in docker.env be exported:
```
set -a; source /appsmith-stacks/configuration/docker.env; set +a
```

arminus · March 25, 2023, 10:29am

I honestly don’t understand your answer…

When starting my appsmith container with docker compose, I set

    environment:
      APPSMITH_MONGODB_URI: mongodb://mongodb/appsmith

I patched that container’s entrypoint.sh (again, latest git) like so

echo "Checking Replica Set of external MongoDB at $APPSMITH_MONGODB_URI ..."

and during startup it says

2023-03-25T10:03:26.356516832Z Checking Replica Set of external MongoDB at mongodb://mongodb/appsmith ...
2023-03-25T10:03:26.356557183Z + echo 'Checking Replica Set of external MongoDB at mongodb://mongodb/appsmith ...'
2023-03-25T10:03:26.356600590Z + appsmithctl check-replica-set
2023-03-25T10:03:58.485667369Z Error trying to check replicaset MongoServerSelectionError: connect ECONNREFUSED 127.0.0.1:27017
2023-03-25T10:03:58.485769368Z     at Timeout._onTimeout (/opt/appsmith/utils/node_modules/mongodb/lib/sdam/topology.js:312:38)
...

However, as it turns out, this message referencing 127.0.0.1 is misleading (I guess it is bening produced by the mongodb container itself), the problem is apparently a timing issue and only happens if mongodb is started together with appsmith.

i.e. if I start mongodb separately, wait a bit and then start appsmith all is fine.

depends_on: mongodb for the appsmith container does not help

sharat87 · March 29, 2023, 7:44am

Thanks. I didn’t realize you’re setting the APPSMITH_MONGODB_URI in the docker-compose.yml file directly. I would recommend moving it to stacks/configuration/docker.env.

The appsmitctl commands don’t pick from the environment directly. They take values from the docker.env. This is what’s causing the confusion for you, I’ll take this up as a bug.

If you change the MongoDB URI in docker.env as well, the check-replica-set command should start working for you.

Regarding depends_on not working, I think you also need to add a healthcheck to your MongoDB container, something like this:

  mongo:
    image: <mongo-image>
    healthcheck:
      test: ["CMD", "mongo ", "--eval 'db.runCommand({ping:1})'"]
      interval: 10s
      timeout: 3s
      retries: 20