Organisation mirror not working

[CrazyWolf13]

Hi @jaedle
since the recent update yesterday, it seems to not correctly apply env vars, I started a container with:

docker run -dit \
  --name github-forgejo-mirror \
  --restart unless-stopped \
  -e GITHUB_USERNAME="CrazyWolf13" \
  -e GITEA_URL=<GITEA_URL> \
  -e GITEA_TOKEN=<Forgejo-Token> \
  -e GITHUB_TOKEN=<Github-Token> \
  -e MIRROR_ORGANIZATIONS=true \
  -e INCLUDE_ORGS="ProxmoxVE" \
  -e EXCLUDE_ORGS="SomeOrg" \
  -e PRESERVE_ORG_STRUCTURE=true \
  -e SINGLE_REPO="" \
  -e GITEA_ORGANIZATION="CrazyWolf13-Mirror" \
  -e MIRROR_PRIVATE_REPOSITORIES=true \
  jaedle/mirror-to-gitea:latest

startup log:

Starting to create mirrors...
2025-04-08T09:08:41.002Z INF config={"github":{"username":"CrazyWolf13","token":"[Redacted]","skipForks":false,"privateRepositories":true,"mirrorIssues":false,"mirrorStarred":false,"mirrorOrganizations":true,"useSpecificUser":false,"includeOrgs":["ProxmoxVE"],"excludeOrgs":["SomeOrg"],"preserveOrgStructure":true,"skipStarredIssues":false},"gitea":{"url":"XXXXX","token":"[Redacted]","organization":"CrazyWolf13-mirror","visibility":"public","starredReposOrg":"github"},"dryRun":false,"delay":3600,"include":["*"],"exclude":[""],"singleRun":false} msg=applied configuration
Organization CrazyWolf13-mirror already exists
Processing repositories from 0 organizations
Found 12 repositories to mirror
Repository Chameleons-Beauty is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository dashy is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository auto_gfpgan-installer is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository CrazyWolf13 is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository grade-store is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository ProxmoxVE is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository home-assets is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository dotfiles is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository chezmoi is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository streamlink-webui is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository web-check is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Repository unix-pwsh is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Waiting for 3600 seconds...

So I expect this to mirror the following:

All my personal repos e.g. https://github.com/CrazyWolf13?tab=repositories --> to the CrazyWolf13-mirror on Gitea, seems to work
Mirror Single_Repo to CrazyWolf13-mirror, works.
Mirror the Org community-scripts for example, which I'm member of and all repos to a new Org called community-scripts on Gitea --> NOT WORKING
Mirror ProxmoxVE, which I'm not a Member, but is in additional to a new Org called ProxmoxVE on Gitea. --> NOT WORKING

Do I need some sort of special permissions on my github_pat?
I added the following:

Thanks!

EDIT: I allow myself to ping @arunavo4 as he is the author of the PR and maybe I understood something wrong about the functionality.

[jaedle]

Thanks for raising the issue and sorry for the (potential) bug.

I will have a look this evening.

[arunavo4]

@CrazyWolf13 Use this endpoint to check if your PAT token has the proper access

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/orgs/<ORG_NAME>/personal-access-tokens/<PAT_TOKEN>/repositories

Github API Docs

[CrazyWolf13]

@arunavo4
Thanks for the quick reply!

{
  "message": "Resource not accessible by personal access token",
  "documentation_url": "https://docs.github.com/rest/orgs/personal-access-tokens#list-repositories-a-fine-grained-personal-access-token-has-access-to",
  "status": "403"
}

I've just tested the above, and my PAT where my account is the owner did not work, and a PAT where I made the org community-scripts ower did not work. I added the following permissions: read-access to code, commits, statuses and metadata.

Also wouldn't that setup then require two PATs, one to get my personal repos and then one per org, which at the moment seems not to be possible in this project?

[arunavo4]

@CrazyWolf13

Also wouldn't that setup then require two PATs, one to get my personal repos and then one per org, which at the moment seems not to be possible in this project?

I think we will have to switch to this, will test this setup and make a PR.

Edit: Its a bug will fix it

[CrazyWolf13]

@arunavo4
Why is this even required?

In my eyes it shouldn't require a per-org access token, if I just want public repos of an org, only if I want the private ones too right?

Also back to the original issue, shouldn't then the ProxmoxVE org be backed up too?

In the current state the project didn't mirror a single repo from any org, not from repos where my GH User is member, nor any of the additional orgs to be backed up.

[arunavo4]

@CrazyWolf13 Yes this seems to be a bug as its public repos should be accessible. Will make a PR for this and fix this issue.

For private you can use the same PAT token but you need to turn on this option.

[jaedle]

@CrazyWolf13 as a quick fix I have just released a docker image with the last working version for your use-case v1.5.0. Feel free to downgrade until this is finally resolved.

Thanks for raising the issue 🙏

[CrazyWolf13]

@jaedle

Thanks a lot for looking into this!
Really appreciate this app!

Sadly I don't see any real change with the 1.5.0 tag, with the exact same env variables :

Starting to create mirrors...
Starting with the following configuration:
 - GITHUB_USERNAME: CrazyWolf13
 - GITHUB_TOKEN: ****
 - MIRROR_PRIVATE_REPOSITORIES: true
 - GITEA_URL: <SERVER_URL>
 - GITEA_TOKEN: ****
 - SKIP_FORKS: false
 - DRY_RUN: false
 - INCLUDE: *
 - EXCLUDE: 
Found 12 repositories on github
Repository is already mirrored; doing nothing:  CrazyWolf13
<..........>
Repository is already mirrored; doing nothing:  web-check
Waiting for 3600 seconds...

The output just looks different, and all the organisation stuffs seems to be removed again? Now mirroring my users repositorys seems to be working, however this worked before too.

[arunavo4]

@CrazyWolf13 @jaedle
PR #34

#!/bin/bash
# Docker test commands for mirror-to-gitea

# Replace these variables with your actual values
GITHUB_USERNAME="your-github-username"
GITEA_URL="http://your-gitea-instance:3000/"
GITEA_TOKEN="your-gitea-token"
GITHUB_TOKEN="your-github-token"

# Basic test with dry run mode
echo "Running basic test with dry run..."
docker run -it --rm \
  -e GITHUB_USERNAME="$GITHUB_USERNAME" \
  -e GITEA_URL="$GITEA_URL" \
  -e GITEA_TOKEN="$GITEA_TOKEN" \
  -e GITHUB_TOKEN="$GITHUB_TOKEN" \
  -e MIRROR_ORGANIZATIONS=true \
  -e DRY_RUN=true \
  arunavo4/mirror-to-gitea:latest

# Test organization mirroring only (no personal repos)
echo "Running organization mirroring test..."
docker run -it --rm \
  -e GITHUB_USERNAME="$GITHUB_USERNAME" \
  -e GITEA_URL="$GITEA_URL" \
  -e GITEA_TOKEN="$GITEA_TOKEN" \
  -e GITHUB_TOKEN="$GITHUB_TOKEN" \
  -e MIRROR_ORGANIZATIONS=true \
  -e ONLY_MIRROR_ORGS=true \
  -e PRESERVE_ORG_STRUCTURE=true \
  -e DRY_RUN=true \
  arunavo4/mirror-to-gitea:latest

# Test with specific organizations included
echo "Running test with specific organizations..."
docker run -it --rm \
  -e GITHUB_USERNAME="$GITHUB_USERNAME" \
  -e GITEA_URL="$GITEA_URL" \
  -e GITEA_TOKEN="$GITEA_TOKEN" \
  -e GITHUB_TOKEN="$GITHUB_TOKEN" \
  -e MIRROR_ORGANIZATIONS=true \
  -e INCLUDE_ORGS="org1,org2" \
  -e PRESERVE_ORG_STRUCTURE=true \
  -e DRY_RUN=true \
  arunavo4/mirror-to-gitea:latest

# Uncomment to run with actual mirroring (no dry run)
# echo "Running actual mirroring..."
# docker run -it --rm \
#   -e GITHUB_USERNAME="$GITHUB_USERNAME" \
#   -e GITEA_URL="$GITEA_URL" \
#   -e GITEA_TOKEN="$GITEA_TOKEN" \
#   -e GITHUB_TOKEN="$GITHUB_TOKEN" \
#   -e MIRROR_ORGANIZATIONS=true \
#   -e ONLY_MIRROR_ORGS=true \
#   -e PRESERVE_ORG_STRUCTURE=true \
#   -e DRY_RUN=false \
#   arunavo4/mirror-to-gitea:latest

# Uncomment to run as a background service
# echo "Running as background service..."
# docker run -d --name mirror-to-gitea \
#   -e GITHUB_USERNAME="$GITHUB_USERNAME" \
#   -e GITEA_URL="$GITEA_URL" \
#   -e GITEA_TOKEN="$GITEA_TOKEN" \
#   -e GITHUB_TOKEN="$GITHUB_TOKEN" \
#   -e MIRROR_ORGANIZATIONS=true \
#   -e ONLY_MIRROR_ORGS=true \
#   -e PRESERVE_ORG_STRUCTURE=true \
#   -e DRY_RUN=false \
#   arunavo4/mirror-to-gitea:latest

[arunavo4]

The README now also contains how to fix any issues and the log also shows why it fails in mirroring orgs if it fails

[CrazyWolf13]

@arunavo4 Thanks!

Quickly pulled your iamge, however in the docker logs I only see:

exec /sbin/tini: exec format error
exec /sbin/tini: exec format error
exec /sbin/tini: exec format error
exec /sbin/tini: exec format error

[CrazyWolf13]

Ah got it, it's only available for arm, I have amd_64.

[arunavo4]

@CrazyWolf13 Sorry My bad i pushed only from my macbook so there is only arm image. Try again I have added build for other platforms also now.

[CrazyWolf13]

@arunavo4 Thanks!

Just deployed, and it seemed to create the org ProxmoxVE (that however was the wrong org, which contains nothing: https://github.com/ProxmoxVE )
Using the org Proxmox, no org was created and nothing mirrored: https://github.com/Proxmox

Starting to create mirrors...
2025-04-15T06:55:18.287Z INF config={"github":{"username":"CrazyWolf13","token":"[Redacted]","skipForks":false,"privateRepositories":true,"mirrorIssues":false,"mirrorStarred":false,"mirrorOrganizations":true,"onlyMirrorOrgs":false,"useSpecificUser":false,"includeOrgs":["Proxmox"],"excludeOrgs":[],"preserveOrgStructure":true,"skipStarredIssues":false},"gitea":{"url":"http://10.10.20.30:3000","token":"[Redacted]","organization":"CrazyWolf13-mirror","visibility":"public","starredReposOrg":"github"},"dryRun":false,"delay":3600,"include":["*"],"exclude":[""],"singleRun":false} msg=applied configuration
Organization CrazyWolf13-mirror already exists
Using authenticated user endpoint to fetch organizations
Direct API call response status: 200
Direct API call found 0 organizations
Paginated API call found 0 organizations
No organizations found through standard endpoints. Trying direct API calls to specific organizations.
Successfully found organization: Proxmox
Found 1 organizations: proxmox
Filtering to include only these organizations: Proxmox
Processing repositories from 0 organizations: 
No organizations to process after filtering. Check your INCLUDE_ORGS and EXCLUDE_ORGS settings.
Found 12 repositories to mirror
Fetching GitHub organizations for mirroring...
Using authenticated user endpoint to fetch organizations
Direct API call response status: 200
Direct API call found 0 organizations
Paginated API call found 0 organizations
Using public endpoint to fetch organizations for user: CrazyWolf13
Direct public API call response status: 200
Direct public API call found 0 organizations
No organizations found through standard endpoints. Trying direct API calls to specific organizations.
Successfully found organization: Proxmox
Found 1 organizations: proxmox
Filtering to include only these organizations: Proxmox
Found 0 GitHub organizations to mirror: 
No organizations to mirror after filtering. Check your INCLUDE_ORGS and EXCLUDE_ORGS settings.
Repository auto_gfpgan-installer is already mirrored in organization CrazyWolf13-mirror; doing nothing.
<......>
Repository chezmoi is already mirrored in organization CrazyWolf13-mirror; doing nothing.
Waiting for 3600 seconds...

I expect the following:

• community-scripts org should definitely be mirrored, as I'm member and it's public.
• Proxmox Org should be mirrored with it's 74 repos, however not even the org was created.