9
Issues hosting Lemmy (via ansible) (lemmy.world)
submitted 3 years ago* (last edited 3 years ago) by to c/selfhosted@lemmy.world
18 comments fedilink hide all child comments
 

(Using https://github.com/LemmyNet/lemmy-ansible)

The ansible-playbook command itself connected to the VPS and completed without any errors or warnings. This is on a completely fresh VPS with Ubuntu Server 22.04.

I created all directories needed in the guide, and the only file I modified was the inventory/hosts file - filling in the username/domain for SSH, domain name, contact email and adding the location of the private key for SSH.

The guide didn't note any changes to config.hjson I needed to make, so I copied that file as requested but left it with the default content. I'm thinking if I missed something it's most likely there.

I couldn't access the web UI and with some investigation I found the dessalines/lemmy:0.17.4 (backend) container is continually restarting, apparently because it can't reach the database - sudo docker container logs <id> returns the following:

thread 'main' panicked at 'Error connecting to postgres://lemmy:PasswordRedacted@postgres:5432/lemmy', crates/db_schema/src/utils.rs:161:56

I'm not sure what to do at this point, so I would be very appreciative of any help with this issue.

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 3 years ago (1 child)

suggest you check postgres is running, accepting connections and returning results on the container first:

docker exec -it lemmyname_postgres_1 /bin/bash

assuming you can connect to the docker container and see a prompt, does this command put you at a sql prompt?

psql -U lemmy -d lemmy

if so, let's confirm that a sql command works.

select name from person; (will likely be an empty set if you haven't set up an admin user yet).

Does all this work OK?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 0 points 3 years ago (1 child)

    docker exec -it lemmyname_postgres_1 /bin/bash returns the bash prompt as expected: 2ab8c789fdcd:/#

    psql -U lemmy -d lemmy successfully opens psql:

    psql (15.3)
Type "help" for help.

lemmy=#

    ran select name from person; from the psql shell, it does look empty (haven't created any users yet):

    
ERROR:  relation "person" does not exist
LINE 1: select name from person;
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 years ago (1 child)

    it should return an empty set. It looks like the database is running but the lemmy database tables were not installed. Can you get back to the sql prompt and try

    \dt

    if the database was installed correctly, you should see something like this:

    lemmy-# \dt

                  List of relations

    Schema | Name | Type | Owner

    --------+----------------------------+-------+-------

    public | __diesel_schema_migrations | table | lemmy

    public | activity | table | lemmy . .

    . .

    public | person | table | lemmy

    . . .

    can you check that the tables are actually there? My guess is that the tables in the database weren't created. assuming you don't see any tables when you do this, again at the sql prompt, please try

    \c lemmy

    and see if it connects? if it does, it will say You are now connected to database "lemmy" as user "lemmy".

    Let me know what the result of that is and we'll try to figure out what's gone wrong with the database setup because it looks like that's your problem - the database was not populated when it was created.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 0 points 3 years ago* (last edited 3 years ago) (1 child)

    \c lemmy does return "You are now connected to database "lemmy" as user "lemmy""

    Running \dt after connecting returns "Did not find any relations", so it looks like you were right and the database didn't get populated.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 years ago (1 child)

    ok thanks - I'm not very familiar with the lemmy installation process so I will need to check if possibly the schema objects in the database aren't created until the admin user is first set up, which is another possibility, but they definitely aren't there right now - I'll set up a fresh install to check in a few minutes.

    Switching gears a bit - when you say you can't access the web UI, can you elaborate on that? The web server should be running. You are going to https://your.url, presumably and then what do you actually see in the browser? It should show an initial setup screen - do you not see anything or is there some kind of error message? When the ansible script runs to create the instance does it throw any errors at all or everything looks happy in the output, including creation of your ssl certs?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 0 points 3 years ago* (last edited 3 years ago) (1 child)

    No errors of any kind in the ansible script. nginx is running, and it's at https://lemmy.name - I get some gateway timeout errors once in a while, but that's all. When I do get error pages, they are served via https at least, so I don't think there's anything wrong with that part.

    I assume this is because the lemmy backend container is continually restarting (as I detailed more in my original post and other replies here) due to database connection errors

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 years ago (1 child)

    Ok so I ran a clean install and confirmed that the database objects exist. So that's definitely the problem. Your database is munted. An interesting thing happened when I re-ran the playbook - I noticed it didn't rebuild the postgres container. Very suspicious! Let's try blowing it away completely and re-running your script:

    docker stop lemmyname_postgres_1

    docker rm lemmyname_postgres_1

    now re run the ansible playbook and let's see if that works.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 0 points 3 years ago* (last edited 3 years ago) (2 children)

    Ran those commands successfully, then the playbook again. Playbook has zero errors or warnings.

    The postgres container was remade.. with the same broken database. Went into it with psql like before, did \c lemmy, then \dt - same error - did not find any relations.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 3 years ago (1 child)

    This alternative installer might be a way around this issue: https://github.com/ubergeek77/Lemmy-Easy-Deploy

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 0 points 3 years ago* (last edited 3 years ago) (1 child)

    Wiped the VPS clean, new 22.04 server install.

    Deployed using that installer.

    Exact same issue. The lemmy container cannot reach the database and continually restarts. I'll post some logs, but they're practically identical to what I posted before:

    lemmy container logs:

    `thread 'main' panicked at 'Error connecting to postgres://lemmy:PasswordRedacted@postgres:5432/lemmy', crates/db_schema/src/utils.rs:161:56

    sudo docker ps output:

    CONTAINER ID   IMAGE                        COMMAND                  CREATED          STATUS
 PORTS                                                                                         NAMES
9eaa62ea6f99   caddy:latest                 "caddy run --config …"   12 minutes ago   Up 12 minutes
 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 443/udp, 2019/tcp   lemmy-easy-deploy-proxy-1
a10009ed128d   dessalines/lemmy-ui:0.17.4   "docker-entrypoint.s…"   12 minutes ago   Up 12 minutes
                                                                                               lemmy-easy-deploy-lemmy-ui-1
a166b1486d51   dessalines/lemmy:0.17.4      "/app/lemmy"             12 minutes ago   Restarting (101) 41 seconds ago                                                                                                 lemmy-easy-deploy-lemmy-1
08329653260b   asonix/pictrs:0.3.1          "/sbin/tini -- /usr/…"   12 minutes ago   Up 12 minutes
 6669/tcp, 8080/tcp                                                                            lemmy-easy-deploy-pictrs-1
51af8f312511   postgres:15-alpine           "docker-entrypoint.s…"   12 minutes ago   Up 12 minutes
lemmy-easy-deploy-postgres-1
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 years ago

    well, that rules out your local system and ansible, at least. It seems like there's an inability for the docker containers to reach postgres definitely at the install and probably at runtime - can we check this?

    first let's make sure the postgres container is on the network and has an address:

    sudo docker exec -it <yourinstance>_postgres_1 /bin/sh

    once you're on the container

    ifconfig

    which should return on the second line: inet addr:172.18.0.4 (or similar ipV4 address)

    exit to return to the host and let's test connecting from the host :

    nc -zv 172.18.0.4 5432 (use IP from the DB container)

    Ubuntu host should respond something like : Connection to 172.18.0.4 5432 port [tcp/postgresql] succeeded!

    now let's try the containers

    sudo docker exec -it <yourinstance>_lemmy-ui_1 /bin/sh

    now from that container test connectivity to postgres:

    nc -zv 172.18.0.4 5432 (use IP from the DB container)

    now the other

    sudo docker exec -it <yourinstance>_lemmy_1 /bin/sh

    on the docker containers the nc command should return something like

    172.18.0.4 (172.18.0.4:5432) open

    is that working?

  • permalink
  • fedilink
  • source
  • parent