Self hosted update from 1.7.14 to latest (1.9.7) fails with 503/Appsmith server is unavailable

Hi,

I haven’t updated for a while and was still running 1.7.14, (self hosted) wanted to update to 1.9.7 today,
but receive 503 /Appsmith server is unavailable when restarting appsmith.

I updated first the 1.9.2 checkpoint version and then to 1.9.7

From the log file I can see that Mongock started the data migration
then
[2023-02-13 09:36:41,652] - FAILED OVER - {“id”=“make-applications-public”, “type”=“execution”, “author”=“”, “class”=“DatabaseChangelog2”, “method”=“makeApplicationsPublic”}
[2023-02-13 09:36:41,660] - Mongock migration aborted and DB transaction not enabled. Starting manual rollback process
[2023-02-13 09:36:41,661] - ChangeSet[make-applications-public] does not provide rollback method
[2023-02-13 09:36:41,666] - Mongock has finished
[2023-02-13 09:36:41,666] - Error in mongock process. ABORTED MIGRATION
[2023-02-13 09:36:45,909] - Application run failed

Any idea how to update my instance?

can you please share the full logs?

sure @pranav

I can see following error -

[2023-02-13 09:36:41,666]  - Error in mongock process. ABORTED MIGRATION
io.mongock.api.exception.MongockException: Error in method[DatabaseChangelog2.makeApplicationsPublic] : Unable to invoke Cipher due to bad padding

This usually means that in your docker.env,
APPSMITH_ENCRYPTION_SALT or APPSMITH_ENCRYPTION_PASSWORD property value has extra space in front or at the end. Can you please verify if thats the case?

no, there is no extra space

any other idea how to migrate up to the new version ?
@pranav

@pranav any suggestions?

Hello @passana! Sorry for the delay. Could you please update first to v1.9.2, which is the checkpoint version, and then to the latest version. Please follow the steps from this guide. Let us know if this works for you.

Thanks @Amelia , I tried this already two times. Did not work. The lor I sent you was already after updating to checkpoint version v1.9.2.

Maybe I could update first to a v1.8.x version ? and then to v1.9.2 and then to the actual one ?
What do you think?

Hey! I’ve asked the team about this. Please be patient for a reply. Also can you tell me what platform you’re deploying on? Is it docker, k8s, etc.?

ok, thanks @Laguna

I deployed on Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-137-generic x86_64) with docker
Ddocker version 20.10.12, build e91ed57

Could you please share logs from when you migrated to 1.9.2? The one you shared is from 1.9.7 and it only shows the cipher failure. We would like to make sure of whether there were any failures during the 1.9.2 migration.

HI @Amelia, I updated to checkpoint 1.9.2 and already get a “503 Appsmith server is unavailable”…

Hello Eduard!
Are you using the built-in internal MongoDB or an external/custom MongoDB?

Can you please also confirm if you have copied the encryption password and salt to the new environment after you have backed up the data? If you did not, you need to take these values from the stacks/configuration/docker.env of the old instance and copy them to the new instance in the same file, as per this documentation about backup.

Thank @Amelia, but I did not backup/recover. I migrated only.

I was able to fix the issue:

I slowly migrated, first to 1.8.1, then to 1.8.10 , then to 1.8.15, then to 1.9,1, then to 1.9.2 … and tested each version if it works.

As 1.9.2 worked fine, I updated to current version (1.9.12) and it still works fine.

I think the missing step was migrating to 1.9.1 as this seems to be an important update.
(1.9.0 was not available
ERROR: manifest for appsmith/appsmith-ce:v1.9.0 not found: manifest unknown: manifest unknown)

in the release list you state “checkpoint update” for 1.9.0, not 1.9.2, please check here:

Hey @passana ! Is it possible for you to share the logs from when you upgraded to v1.9.1 as well as v1.9.2? This will help us figure out exactly what is happening with this upgrade and hopefully help out other users that find themselves in your position faster. :slight_smile:

I am nor at v1.9.12. How do I get the logs from the migration of 1.8.15 to 1.9.1 ?

This is our guide on how to get the logs.

oh. so I just need to send all the logs or any specific?

Is there confidential/private data in the logs? Do I need to check for this before submitting it?

Please send all the logs. If you are concerned about confidential data, it’s best to send it to us per email to support@appsmith.com and add the link to this post as reference.