ns8-imapsync

Install

Instantiate the module with:

add-module imapsync

The output of the command will return the instance name. Output example:

{"module_id": "imapsync1", "image_name": "imapsync", "image_url": "ghcr.io/nethserver/imapsync:1.0.5"}

Configure

Let's assume that the imapsync instance is named imapsync1.

We need to bind imapsync to a mail server inside the cluster, we use the MODULE_UUID of mail server to configure imapsync, reveal the vmail master secret of local users and start the container waiting for cron tasks or individual user tasks

Launch configure-module, by setting the following parameters:

• mail_server: module uuid of the mail server
• mail_host: local ip of the mail server

Example:

api-cli run module/imapsync1/configure-module --data '{
    "mail_server":"8dd3b3fe-609c-42f7-a2d1-cecba9461bea",
    "mail_host":"10.5.4.1"
}'

The above command will start and configure the imapsync instance.

start to sync a remote imap account to local user account

Example:

api-cli run module/imapsync1/create-task --data '{
    "localuser": "administrator",
    "remotehostname": "imap.foo.com",
    "remoteport": 143,
    "security": "tls",
    "delete_local": false,
    "delete_remote": false,
    "delete_remote_older": 0,
    "exclude": "folder1,folder2",
    "remoteusername": "username",
    "remotepassword": "password",
    "cron": "5m",
    "task_id": "a0241w",
    "foldersynchronization": "all"
}'

Field descriptions:

• security: tls, ssl, or "" (empty for no security)

• delete_local: true or false – if true, deletes mails and folders on the local account that are not in the remote account.

• delete_remote: true or false – if true, deletes mails on the remote account after synchronization to the local account.

• delete_remote_older: Number of days after which remote emails are deleted. If set to 0, messages are removed after the transfer.

• exclude: Comma-separated list of folders to exclude (e.g., folder1,folder2,^folder3$). Use ^ to mark folder names that must start with a pattern and $ for those that must end with one.

• cron: Defines task scheduling (e.g., 5m runs every 5 minutes, "2h" runs every 2 hours). Internally converted to crontab format (0/5 * * * * for minutes, 1 */2 * * * for hours).

• remote credentials: Provide the username and password of the remote IMAP account. Alternatively, use the IMAP master administrator credentials (e.g., user*vmail with the corresponding password).

• foldersynchronization: Controls which folders are synchronized:

  • all – synchronizes all folders.
  • inbox – synchronizes only the inbox.
  • exclusion – synchronizes all folders except excluded ones.

• task_id: Unique identifier for the task, automatically generated by the UI.

• sieve_enabled: Boolean (true or false). If true, applies the user's Sieve filters to synchronized messages. This option is only considered when foldersynchronization is set to inbox.

delete env and stop a running synchronisation

Example:

api-cli run module/imapsync1/delete-task --data '{
    "localuser":"administrator"
}'

stop a running synchronisation

Example:

api-cli run module/imapsync1/stop-task --data '{
    "localuser":"administrator"
}'

start a running synchronisation

Example:

api-cli run module/imapsync1/start-task --data '{
    "localuser":"administrator"
}'

start all configured tasks

Read all imapsync/*.env files and start the synchronization

Example:

api-cli run module/imapsync1/start-all-tasks

stop all configured tasks

Read all imapsync/*.env files and stop the synchronization

Example:

api-cli run module/imapsync1/stop-all-tasks

read configuration

Example:

api-cli run module/imapsync1/get-configuration

Answer:

{
  "mail_server": "e8a6177c-9ae5-4356-826b-0a5f93b2dbaf",
  "mail_host": "10.5.4.1",
  "mail_server_URL": [
    {
      "name": "mail2",
      "label": "mail2 (R3.rocky9-3.org)",
      "value": "e8a6177c-9ae5-4356-826b-0a5f93b2dbaf,10.5.4.1"
    }
  ],
}

list-tasks

Read configuration from environment and from imapsync/*.{env,pwd}

api-cli run module/imapsync1/list-tasks

{
  "enabled_mailboxes": [
    {
      "name": "administrator",
      "label": "administrator",
      "value": "administrator"
    },
    {
      "name": "foo",
      "label": "foo",
      "value": "foo"
    },
    {
      "name": "john",
      "label": "john",
      "value": "john"
    }
  ],
  "user_properties": [
    {
      "task_id": "qu1dcb",
      "localuser": "administrator",
      "remoteusername": "user2@domain.com",
      "remotehostname": "imap.domain.com",
      "remoteport": "143",
      "security": "tls",
      "delete_local": false,
      "deletefolder": "",
      "exclude": "",
      "delete_remote": false,
      "delete_remote_older": 0,
      "sieve_enabled": false,
      "expunge_remote": "",
      "cron": "",
      "folder_inbox": "",
      "foldersynchronization": "all",
      "remotepassword": "password",
      "service_running": false
    },
    {
      "task_id": "vd2am3",
      "localuser": "john",
      "remoteusername": "user@domain.com",
      "remotehostname": "imap.domain.com",
      "remoteport": "143",
      "security": "tls",
      "delete_local": false,
      "deletefolder": "",
      "exclude": "",
      "delete_remote": false,
      "delete_remote_older": 0,
      "sieve_enabled": false,
      "expunge_remote": "",
      "cron": "",
      "folder_inbox": "",
      "foldersynchronization": "all",
      "remotepassword": "password",
      "service_running": false
    }
  ]
}

list-informations

You can retrieve the email and folder numbers and the email size for the local and remote account Example:

api-cli run module/imapsync9/list-informations --data '{
  "localuser": "john",
  "task_id": "vd2am3"
  }'

output:

{"status": true, "host1Folders": 78, "host2Folders": 78, "host1Messages": 164625, "host2Messages": 155, "host1Sizes": 3060488650, "host2Sizes": 72036894}

in case of error you will have

{"status": false}

custom scripts

If necessary, you can add your custom scripts inside the container. To facilitate this, two volumes, included in the backup, are mounted from their respective directories under the ./state module.

• cron (e.g. ~imapsync1/.config/state/cron) This folder is designated for storing cron files, each of which should end its name with .custom, while adhering to the cron syntax.

  1 */1 * * *  root /usr/bin/imapsync --host1 SOURCE_SERVER --user1 SOURCE_USERNAME --password1 SOURCE_PASSWORD --host2 DESTINATION_SERVER --user2 DESTINATION_USERNAME --password2 DESTINATION_PASSWORD
  

  Instead of directly add an imapsync command, you have the option to run a script, which can be stored in the imapsync volume.

  1 */1 * * *  root /etc/imapsync/script.custom
  

• imapsync (e.g. ~imapsync1/.config/state/imapsync) In this location, you can store custom scripts (ending with .custom) intended for execution by cron. These scripts must be set to be executable by all users.

troubleshot issues

The state/imapsync folder contains the following files:

• Environment files (*.env) – Stores task-related environment variables.
• Password files (*.pwd) – Contains credentials. The special vmail.pwd file holds the Dovecot master user's credentials.
• Lock files (*.lock) – Created when a task is running to prevent multiple simultaneous executions.
• Log files (*.log) – Stores the last Imapsync run log. This file is overwritten with each task execution. A summary, including the number of transferred messages and the Imapsync exit code, is always sent to the system log.

You can filter these files by task name, which is a combination of the localuser and the task_id.

Example of files for two tasks, both for local user foo. One of them is running, the other is stopped:

.config/state/
├── cron
│   ├── foo_a0241w.cron
│   └── foo_a1j44r.cron
└── imapsync
    ├── foo_a0241w.env
    ├── foo_a0241w.lock
    ├── foo_a0241w.pwd
    ├── foo_a0241w.log
    ├── foo_a1j44r.env
    ├── foo_a1j44r.pwd
    ├── foo_a1j44r.log
    └── vmail.pwd

to launch manually a task

runagent -m imapsync1
podman exec -ti imapsync /usr/local/bin/syncctl start foo_a1j44r
podman exec -ti imapsync /usr/local/bin/syncctl stop foo_a1j44r
podman exec -ti imapsync /usr/local/bin/syncctl status foo_a1j44r

Bulk import tasks from CSV

Create multiple synchronization tasks at once using a CSV file with the import-csv-tasks utility script.

Overview

This Python script automates the bulk creation of IMAP synchronization tasks by reading user data from a CSV file and calling the create-task API endpoint for each entry. It's ideal for migrating multiple email accounts from a remote IMAP server to your local mail system.

CSV File Format

The CSV file must contain a header row with these 6 required columns:

This users.csv file can be found at .config/examples/users.csv

runagent -m imapsync1
cat ../examples/users.csv

localusername,remoteusername,remotepassword,remotehostname,remoteport,security
pansy.dumbledore5,user1@example.org,"enquotedPasswordIfSeparatorInside",imap.example.org,993,ssl
lavender.umbridge7,user2@example.org,"enquotedPasswordIfSeparatorInside",imap.example.org,993,ssl
dolores.slughorn3,user3@example.org,"enquotedPasswordIfSeparatorInside",imap.example.org,143,tls

Required columns:

• localusername – Local user account name (must exist on your mail server)
• remoteusername – Remote IMAP account username/email
• remotepassword – Remote IMAP account password (quote if contains special characters)
• remotehostname – Remote IMAP server hostname or IP
• remoteport – Remote IMAP server port (typically 993 for SSL/TLS, 143 for STARTTLS)
• security – Security protocol: ssl, tls, or empty string "" for no encryption

Important notes:

• Column order does not matter – the script maps columns by header name
• Delimiter is comma (,)
• Passwords with special characters should be quoted: "myP@ss,word"
• All fields are mandatory except security – rows with empty required values will be rejected
• The security field can be empty (for no encryption), but the other 5 fields must contain values
• Empty lines in the CSV are automatically skipped

Usage

The script is designed to be used with NS8 runagent to bulk import synchronization tasks.

Basic usage (file redirection):

runagent -m imapsync1 import-csv-tasks < users.csv

Alternative (using pipe):

cat users.csv | runagent -m imapsync1 import-csv-tasks

Check CSV format before creating tasks (check-only mode):

runagent -m imapsync1 import-csv-tasks -c < users.csv

This performs comprehensive validation without making API calls:

• Checks CSV structure (required columns present)
• Validates delimiter (comma-separated)
• Verifies all mandatory fields have values (except security which can be empty)
• Validates port numbers are numeric
• Reports specific errors for each invalid row

Example validation output:

📋 CSV Column Validation:
   Delimiter: ',' (comma-separated)
   Found 6 column(s): localusername, remotehostname, remotepassword, remoteport, remoteusername, security
   Column order: does not matter (mapped by header name)
✓ All 6 required columns present
✓ Found 2 data row(s) (empty lines skipped)

🔍 Validating row data...
✓ All 2 rows validated successfully
✓ Generated 2 unique task IDs

✓ CSV file is valid. No tasks were created (check-only mode).

Display help:

runagent -m imapsync1 import-csv-tasks -h

Options

• -c, --check – Check-only mode: validates CSV format without creating tasks
• -h, --help – Display comprehensive help message

Features

• Flexible input methods:

  • Reads from standard input (stdin)
  • Works with file redirection (<), pipes (|), or heredoc
  • Compatible with NS8 runagent for direct module integration
  • No file path arguments needed

• CSV parsing:

  • Comma-separated delimiter (,) - required
  • Validates all 6 required columns are present before processing
  • Handles quoted fields with embedded delimiters
  • Reports missing columns with clear error messages
  • Skips empty lines automatically

• Data validation:

  • Mandatory field checking: ensures 5 required fields are not empty (localusername, remoteusername, remotepassword, remotehostname, remoteport)
  • Optional field: security can be empty (for no encryption)
  • Port validation: ensures remoteport contains only numeric values
  • Check mode (-c flag): validates all rows without creating tasks
  • Clear error messages with row numbers for easy debugging

• Automatic field population:

  • Generates unique random 6-character task ID (lowercase alphanumeric) for each user
  • Auto-fills optional fields with sensible defaults:
    • cron: "" (empty - no automatic scheduling)
    • delete_local: false (preserve local emails)
    • delete_remote: false (preserve remote emails)
    • delete_remote_older: 0 (no age-based deletion)
    • exclude: "" (no folder exclusions)
    • foldersynchronization: "all" (sync all folders)
    • sieve_enabled: false (don't apply Sieve filters)

• Robust error handling:

  • Validates port numbers are numeric
  • Continues processing remaining users if one fails
  • Displays detailed error messages per user
  • Returns non-zero exit code if any task fails

• Progress tracking:

  • Shows real-time progress for each user creation
  • Displays success/failure status per task
  • Provides final summary with success/failure counts

• API integration:

  • Calls action/create-task for each user
  • Passes data as properly formatted JSON
  • Captures and displays API response output

Example Output

Create 3 synchronization tasks from a CSV file:

runagent -m imapsync1 import-csv-tasks < users.csv

Validating CSV file: users.csv

📋 CSV Column Validation:
   Delimiter detected: ';'
   Found 6 column(s): localusername, remotepassword, remotehostname, remoteport, remoteusername, security
✓ All 6 required columns present
✓ Found 3 data row(s)

📦 Starting task creation with module: imapsync1
   Processing 3 user(s)...

Creating task for pansy.dumbledore5 (ID: a1b2c3)...
✓ Success: pansy.dumbledore5

Creating task for lavender.umbridge7 (ID: d4e5f6)...
✓ Success: lavender.umbridge7

Creating task for dolores.slughorn3 (ID: g7h8i9)...
✓ Success: dolores.slughorn3

======================================================================
📊 Summary: 3 successful, 0 failed
======================================================================

Troubleshooting

"Missing required columns"

• Check CSV header row contains all 6 required column names exactly as specified
• Verify no typos in column names (case-sensitive)

"Missing required value(s)"

• The script rejects rows with empty mandatory fields
• Ensure all 5 required fields have values: localusername, remoteusername, remotepassword, remotehostname, remoteport
• Only security can be empty (for no encryption)
• Use -c flag to validate CSV before creating tasks

"Invalid port number"

• Ensure remoteport column contains only numeric values (e.g., 993, 143)

Remove all tasks after a bad import

If you need to clean up after a failed or incorrect bulk import, you can remove all task configuration files:

# Stop all running tasks first
api-cli run module/imapsync1/stop-all-tasks

# Remove all task configuration files (replace imapsync1 with your module ID)
rm -f /home/imapsync1/.config/state/imapsync/*.env
rm -f /home/imapsync1/.config/state/imapsync/*.pwd
rm -f /home/imapsync1/.config/state/imapsync/*.log
rm -f /home/imapsync1/.config/state/imapsync/*.lock
rm -f /home/imapsync1/.config/state/cron/*.cron

# Verify cleanup
ls -la /home/imapsync1/.config/state/imapsync/

Note: This preserves the vmail.pwd file (master password). To start fresh completely, you can also remove it, but you'll need to reconfigure the module afterward.

Uninstall

To uninstall the instance:

remove-module --no-preserve imapsync1

Testing

Test the module using the test-module.sh script:

./test-module.sh <NODE_ADDR> ghcr.io/nethserver/imapsync:latest

The tests are made using Robot Framework

UI translation

Translated with Weblate.

To setup the translation process:

• add GitHub Weblate app to your repository
• add your repository to [hosted.weblate.org]((https://hosted.weblate.org) or ask a NethServer developer to add it to ns8 Weblate project