From ajcody
Jump to: navigation, search

Contents

Backup Next Generation (NG)

This is a copy left for Cine's diff for the adoc. Please review and edit this one: Ajcody-Draft-8.8-Beta-Backup-NG-ZetaForked-adminguide .

Primary Source

Example of Author/Contributor Block - Remove For Prod

Primary Authors Contributors
Cine [Orginal adoc] Ajcody [Editing in Red], Cine [Editing in Blue]

Recommended Resorting of And New Sections

Re-sort Outline

I would re-sort the sections and include a couple of new ones, to end up with something like this. -Ajcody

**Overview & Introduction Topics**

[New Section] Compatibility
[New Section] Comparison of Backup NG And Old NE Backup
[New Section] Prereq and Planning Information/Checks
[New Section] Details on the REDOLOG
Operation queue and queue management
Real Time Scan
Smartscan
doCoherencyCheck
Purge
Details on accounts state during restore/backup operations
Third-Party Clients And Restores - ZCO, ZD, IMAP

** Introduction To The Two Backup Methods **

[New Section] Main Backup Method [directory path info and what not]
COS-level Backup Management
[New Section] Notes About Domain/User options
External Backup
Taking Additional and Offsite backups of Backup NG's Datastore

** Common Restore Methods **

1.xx New Header - Common Restore Methods
1.17 Unrestorable items
1.11 Restore on New Account
1.12 Undelete Restore
1.13 External Restore
1.14 Restore Deleted Account
1.15 Item Restore

** Non-Routine Restore Methods **

[New Section] Non-Routine Restore Methods
[New Section] Certificates
Multistore Specific Information [reworded from "Multistore Informations"]
Disaster Recovery
[split this existing section into two] Incremental migration with Backup NG
[New Section - Split] Incremental migration with Backup NG - First Import All
[New Section - Split] Incremental migration with Backup NG - Provisioning Import Only First

[New Section] Compatibility

New Section : Either here or somewhere else, there should be a compatibility statement about importing/export from one ZCS version to another ZCS version. Also, clarity on the OS. -Ajcody

Too specific for the Admin Guide, but worth adding to any Wiki article about restores (or to a "Restore Hub" page with links to all restore articles) - Cine

[New Section] Comparison of Backup NG And Old NE Backup

New Section - I think it would be nice to offer a birds overview of the changes/diff's between NE Backup and Backup NG. This might help clear up confusion as they read through the rest of the Backup NG manual. Reference, maybe, something like this - http://www.digitalhandshakes.com/wiki/index.php/Ajcody-Draft-8.8-Beta-Backup-NG-And-NE-Backup-Table

Dedicated Wiki article - Cine

[New Section] Certificates

New Section : Either here or somewhere else, there should be a clarification about certificates when relevant. There currently is no reference to them anywhere. -Ajcody

[ACTION] Add certificate info in the Realtime Scan, External Backup, External Restore and Disaster Recovery sections - Cine

[New Section] Prereq and Planning Information/Checks

New Section : Insert before 'managing the real time scanner' a section about prereq' and planning for Backup NG.}}

  • This comment, below, which shows up later should be more fully described in the Preq/Planning section.
    • "Note: when the Real Time Scanner is enabled for the first time or re-enabled after a stop, a Live Full Scan is required. A warning will be displayed after enabling the Real Time Scanner and you will be prompted to start the Full Scan."

There’s no mention of disk space requirements for the backup target directory/partition.

  • No mention of performance ‘expectations’ during “live full scan” the first time it’s run on an existing system.

[ACTION] There is a dedicated "First steps with NG Modules" document in the documentation, IIRC it's in the install guide. I'll update that with the relevant info as well as fix/improve the Realtime Scan section - Cine

[New Section] Details on the REDOLOG

New Section : That discusses in detail about the redolog.

Statement about accounts NOT going into maintenance mode during backup.

Not needed anymore, since there is no more need for archived RedoLogs now. Backup NG will disable RedoLog archiving after being initialized after clearing out any backup-related CRON line. - Cine

[New Section] Details on accounts state during restore/backup operations

New Section : New section that details/describes what does and doesn't happen to accounts during a restore/backup process. I think it's notable to point out in its own section this difference between Backup NG and the old NE Backup.

  • Also, it should be noted either here or later under the ‘restore methods’ that accounts aren’t “deleted” prior to a restore [for the restore method where it’s relevant]

Part of the "Differences between Legacy Backup and Backup NG" article - Cine

[New Section] Main Backup Method

New Section : [directory path info and what not]

Not sure this is worth a dedicated section in the Admin Guide IMHO, since both the RealTime Scanner and the Backup Path have their own sections. Surely an interesting topic for a Wiki article or whitepaper, but not a priority. - Cine

[New Section] [New Section] Notes About Domain/User options

New Section : [Or make one section that briefly touches on COS, Domain, User options for the backup/restore commands]

[ACTION] Split the info in each dedicated Restore section instead, every restore command has different options and those change in time - Cine

[New Section Header] - Common Restore Methods

New Section : [Would include : Restore on New Account , Undelete Restore , External Restore, Restore Deleted Account , Item Restore

Dedicated Wiki article with real world usage examples - Cine

[New Section Header] - Non-Routine Restore Methods

New Section : [Would include : Certificates , Multistore Specific Information [reworded], Disaster Recovery, Incremental migration with Backup NG

Dedicated Wiki article with real world usage examples - Cine

[New Section - Split] Incremental migration with Backup NG - First Import All

New Section : [Would be specific to doing a FULL initial import of data.

[New Section - Split] Incremental migration with Backup NG - Provisioning Import Only First

New Section : [Would be specific to doing a provisioning only import first, the switch over, and then another import with all the data.

[ACTION] Worth mentioning as a common scenarion in the Incremental Migration Guide, I'd rather have one guide with all the info instead of splitting stuff - Cine

[New Section] Third-Party Clients And Restores - ZCO, ZD, IMAP

New Section : Mention details about impact to 3rd party clients when it comes to restores.

Dedicated Wiki article - Cine

Real Time Scan

What is the Real Time Scanner?

The Real Time Scanner is the big innovation in Backup NG. Each event on the system is recorded live to Zimbra's RedoLog and saved by Backup NG, which means that it is always possible to rollback an account to a previous state. Thanks to the Real Time Scanner all the restore modes work with a split-second precision.

How Does it Work?

The Real Time Scanner reads all the events of the mail server almost real-time by following the flow of informations information provided by the RedoLog. Then it replicates the same operations on it's its own data structure, creating items or updating their metadata. No information is ever overwritten in the backup, so that every item has its own complete history.

Managing the Real Time Scanner

Enabling the Real Time Scanner

Via the Administration Zimlet
  • Select the Backup NG Tab.
  • Under Real Time Scanner, press the Enable button.
Note: when When the Real Time Scanner is enabled for the first time or re-enabled after a stop, a Live Full Scan is required. A warning will be displayed after enabling the Real Time Scanner and you will be prompted to start the Full Scan.
Via the CLI

To enable the Real Time Scanner via the CLI, the ZxBackup_RealTimeScanner property of the Backup NG module must be set to true:

zxsuite backup setProperty ZxBackup_RealTimeScanner TRUE

[ Because you mentioned enabling of the 'live full scan' requirement in the zimlet section above, should there also be the CLI example on how to do that here? ] No need to, this section has all the info - Cine

Disabling the Real Time Scanner

Via the Administration Zimlet
  • Select the Backup NG Tab.
  • Under Real Time Scanner, press the Disable button.
Via the CLI

To disable the Real Time Scanner via the CLI, the ZxBackup_RealTimeScanner property of the Backup NG module must be set to false:

zxsuite backup setProperty ZxBackup_RealTimeScanner FALSE

[ Maybe some more clarity on this. I would expect someone to ask, “That disables it in ‘realtime’ or the next ZCS restart or some event where the new variable is read?” ]

[ACTION] Add this to the main RealTime Scanner section, mention that it's enabled and disabled right away without any server/service restart

Why Should I Disable the Real Time Scanner?

The only time you should stop the Real Time Scanner is while performin performing an External Import of multiple domains. This is a safety measure to avoid high load on your server. After the import just , re-enable the Real Time Scanner and perform a SmartScan when prompted , and you are good to go! :

[ Reading from top to bottom, at this point we've now heard of "real time scanner", live full scan, and now smartscan. Just noting, that this might be cleared up as to not confuse readers. ]

Limitations and Safety Scan

[ Now we have a safety scan? Should we just drop that in the title for this section? ]

At the moment t There are two main limitations when restoring data acquired via the Real Time Scanner.

[ You only go over one limitation below, I don’t see a “second” one. ]

Namely, those are

* Emptied Folder - when a user uses the Empty Folder button in the right-click context menu.

[ I would reformat the examples items and put this part underneath the next section, stating it’s an example of an operation which can cause Backup NG not to determine the status of an item/account. ]

In this case, and Any time Backup NG cannot determine the status of an item by reading the metadata saved by the Real Time Scan, an Account Scan on the given account is triggered BEFORE the restore.

This fixes any misaligned data and sanitizes the backed up metadata for the mailbox.

Insert the examples here, at the bottom of section. ]

SmartScan

What is the SmartScan?

[ This section and ‘how does it work’ doesn’t really answer the fundamental question – WHY is SmartScan needed if there’s a REAL TIME Scanner? And without knowing that, the reference of it running once a night leads one to wonder if you might have one day of possible ‘lost data/changes’. ]

The SmartScan is the main coherency check for the health of your backup system. It's Smart because it operates only on accounts modified since the last SmartScan, hence improving system performance and decreasing scan time exponentially.

[ End of doc then mentions "What is the Coherency Check? The Coherency Check is a feature added in Network NG Modules 1.10.2 which performs a deeper check of a Backup Path than the one done by the SmartScan." I would move the Coherency Check section underneath this section. ]

By default, [comma] a SmartScan is scheduled to be executed each night (if Scan Operation Scheduling is enabled in the Backup NG section of the Administration Zimlet). [ What if Backup NG was enabled from the CLI? ] Once a week, on a day set by the user, a Purge is executed together with the SmartScan to clear Backup NG's datastore from any deleted item that exceeded the retention period.

How Does it Work?

The Backup NG engine scans all the items on the Zimbra Ddatastore looking for items modified after the last SmartScan updating. It updates, any outdated entries and creating any item not yet present in the backup [ This is confusing, how is it real-time backup then? ] while flagging as deleted any item found in the backup and not in the Zimbra Ddatastore.

Finally, it updates all Cconfiguration metadata in the backup [ Lacks details ], so that Ddomains, Aaccounts, COSs and Server Configurationsserver configureations are stored.

When is a SmartScan Executed?

  • When the Backup NG module is started.
  • Daily, if the Scan Operation Scheduling is enabled in the Administration Zimlet.
  • When the Real Time Scanner is re-enabled via the Administration Zimlet after being previously disabled.

Running a SmartScan

[ Why would a sys admin run this manually? It was mentioned it's a scheduled task above. ]

Starting the Scan via the Administration Zimlet

  • Open the Administration Zimlet
  • Click the Backup NG tab (be sure to have a valid [ Something is missing here. ]
  • Click Run Smartscan

Starting the Scan via the CLI

To start a FullScan via the CLI the doSmartScan command is available

[ I'll note this once, but in this doc, it uses the syntax where the help out is shown WITHOUT the use of the help flag. It's probably not a big deal, but I would lean towards actually using the help flag within these examples. As to prevent any accidental copy/pastes. ]

Syntax:
   zxsuite backup doSmartScan [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                TYPE
notifications(O)    Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup dosmartscan notifications [email protected],[email protected]
Performs a smart scan and sends notifications to [email protected] and [email protected]

Checking the Status of a Running Scan

To check the status of a running scan via the CLI the monitor command is available:

[ I think it would be useful to state what the possible "status" returns would be. Running, stopped, etc? Just so it's documented. ]

Syntax:
   zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                 TYPE
operation_uuid(M)    Uiid
operation_host(O)    String

(M) == mandatory parameter, (O) == optional parameter

Purge

What is the Backup Purge?

The Backup Purge is a cleanup operation that removes from the Backup Path any deleted item which that exceeded the retention time defined by the Data Retention Policy.

How Does it Work?

The Purge engine scans through the metadata of all deleted items, and removes any item whose last update (deletion) timestamp is higher than the retention time.

Should an item BLOB still be If an item BLOB is still referenced by one or more valid metadata files - due to Backup NG's built-in deduplication - the BLOB itself will not be deleted. [ Might include one more sentence to state what it exactly DOES DO in regards to the users backup data. It would educate admin to understand more what’s going on in the backup directory/structure. ]

Starting from Network NG Modules 1.8.17, Postfix Ccustomizations backed up by Backup NG also follow the backup path's purge policies : t. This can be changed in the Backup NG section of the Administration Zimlet by unchecking the Purge old customizations checkbox. [ If they aren’t purged, will all customizations [postfix only] be kept forever within the backup? ]

When is a Backup Purge Executed?

  • Weekly, if the Scan Operation Scheduling is enabled in the Administration Zimlet. [ Wouldn’t this give a retention period of X a possible X+{up to}7 days? ]
  • When manually started either via the Administration Console or the CLI.

Infinite Retention

Should the Data Retention Policy be set to 0, meaning infinite retention, the Backup Purge will immediately exit [ If running you mean? Or that since it will continue to run weekly [hard coded?] via “scan operation scheduling” it will simply stop upon execution once it identifies retention is set to 0? ] since no deleted item will ever exceed the retention time.

Running a Backup Purge

Starting the Backup Purge via the Administration Zimlet

  • Click the Backup NG tab (be sure to have a valid [ Something is missing here. License? ]
  • Click the Run Purge button on the top-right part of the UI.

Starting the Backup Purge via the CLI

To start a BackupPurge via the CLI the doPurge command is available

Syntax:
   zxsuite backup doPurge [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME              TYPE
purgeDays(O)      String
backup_path(O)    Path

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup dopurge purgeDays 30 backup_path /opt/zimbra/backup/backup_name

Checking the Status of a Running Backup Purge

To check the status [ Running, stopped? What should the sys admin expect as the possible outcome? ] of a running Purge via the CLI, [comma insert] the monitor command is available: [colon insert]

Syntax:
   zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                 TYPE
operation_uuid(M)    Uiid
operation_host(O)    String

(M) == mandatory parameter, (O) == optional parameter

External Backup

What is the External Backup?

The External Backup is one of the Bakcup Backup Modes of Backup NG.

It creates a snapshot [ Snapshot? I would ask, from the moment the command is run or upon its completion? What “date/time” do I use to know when all the accounts were ‘snapped’? ] of the mail system, [ An NE customer with experience of the older modules will go - “mail system, huh?”. What about doing a full backup of just one user or specific users? See zmbackup options. What about employee’s that leave and I need to retain their data for 7 years? ] ready to be used for a migration. Exported data is deduplicated and compressed to optimize disk utilization, transfer times and I/O rates.

[ It might be worth mentioning or adding a small section here stating “what can use” or “how one would later use” an ‘external backup’. ]

How Does it Work?

The Backup NG engine scans through all the data in the Zimbra Ddatastore, saving all the items (deduplicated and compressed) [ Zmbackup had options for this, in a way. How might Backup NGbe different? Worth pointing out? ] on a folder [ Directory, not folder. Windows uses 'folder', Unix/Linux has directories ] of your choice.

Folder [ Directory, not folder. ] Permissions

The destination folder [ Directory, not folder. ] must be readable and writable by the zimbra user.

In order to To create a valid export directory, [comma insert] you can run the following commands: [ Note, your example is assuming root since you are changing the ownership but it’s not mentioned they'll need to be root. [well, depending on where the new dir will be in the filesystem ]

below [ Directory [dir], not folder. ]

mkdir /opt/zimbra/backup/yourdestfolder
chown -R zimbra:zimbra /opt/zimbra/backup/yourdestfolder

[ Above examples, I would probably do something like /your/backup-mnt/backup/export ]

[ Is it worth noting if the export will have a parent directory created under that path and what naming convention it'll use? ]

Preparing the Migration

In order to To minimize the risk of errors, please perform the following mainteinance maintenance procedures before migrating:

  • Doublecheck Zimbra permissions with the following command (must be ran run as root):

[ Note bug on this from 8.7 to 8.7.2 https://bugzilla.zimbra.com/show_bug.cgi?id=106379 ]

[ Hold on. These, zmfixperms & rim & zmblobchk, are big operations. A warning of that might be warranted. ]

/opt/zimbra/libexec/zmfixperms --verbose --extended_
  • Reindex all mailboxes.

[ No example or reference. Doing it for all - how? zmprov doesn't have an option for all. ]

  • Check the BLOB consistency with the zmblobchk utility.

[ No example or reference. ]

[ Why zmblobchk and not zextras hsm/powerstore doCheckBlobs ? ]

Running an External Backup

[ Someone might wonder if they should stop or not their ‘normal’ realtime backup if they are doing an export – or worry about a perf hit, etc ]

Via the Administration Zimlet

[ Full click through should be provided. Don’t assume they know how to get to it within the admin console. Note – this should be double checked for the whole document – that full path for admin console is always included. ]

  • Click the Backup NG tab.
  • Click on the Export Backup button under Import/Export to open the Export Backup wizard.
  • Enter the Destination Path in the textbox and press Next. The software will check if the destination folder is empty and whether the zimbra user has R/W permissions or not.
  • Select the domains you [ This section started with, “ creates a snapshot of the mail system,” and an NE zmbackup user would wonder about “user” backups – now we mention valid targets are “domains”? Note – we don’t say “domains or [all]”. Should be, “the individual domain/s you want or all to export” ] want to export and press Next. [ Another question that will arise is, Q. “Why do they allow me to select normal backups via COS but I can’t use the same COS option for the external domain?” Q. “I only need half of the domain backed up as I have set them up per my COS policy. Q. “Some of these accounts aren’t even in my current backup – what perf hit will those take since now of the data is currently in the backup? ]
  • Verify all your choices in the Operation Summary window. You can also add additional email addresses to be notified when the Rrestore operation is finished. Pleas Please notice that the Admin account and the Uuser who started the Rrestore procedure are notified by default.

Via the CLI

To start an External Backup via the CLI, the doExport command is available:

Syntax:
   zxsuite backup doExport {destination_path} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                   TYPE                  DEFAULT
destination_path(M)    Path
domains(O)             Domain Name[,..]      all
notifications(O)       Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doexport /opt/zimbra/backup/ domains example.com notifications [email protected]
Exports a backup of example.com to /opt/zimbra/backup/ and notifies [email protected]

Scheduling Script

[ What script? ]

The Network NG Modules CLI [ This seem weird like you’re saying they do it via an NG module vs. cron and the cli option to run the ext. backup. ] can be used to schedule [the or a] External Backup operation. This comes very handy This is very useful when you need to keep a daily/weekly/monthly backup for corporate or legal reasons.

[ Why is this here anyway without more details or a working example? This could easily be stated at the start of the ext. backup section. Uses of… ]

[ Note – this section also makes me think, "If I did a script for this to run via cron, do I need to do a new target directory every time?" "What if I don’t, what happens then – a complete overwrite or just the diff’s?" Might be worth answering that somewhere in the doc anyway. ]

Restore on New Account

What is the Restore on New Account?

The Restore on New Account procedure allows you to restore a [Remove a] the contents and preferences of a mailbox as it was in a moment in time, in a completely new account. The source account is not changed in any way, so that is So it's possible to recover one or more deleted items in a user's account without actually rolling back the whole mailbox. [ Why is this even here for the ‘restore on new account” section? This example really applies only to the older zmrestore/NE limitations. With NG/zextras you don’t have to do this to recover deleted items. ] When you run this kind of restore you can choose to hide the newly created account from the GAL as a security measure. [ Why would this be a security issue? Isn’t it about not having a ‘temp/non-production account’ listed in the GAL to be confused with the production one? ]

How does it work?

When a Restore on New Account starts a new account is created (the Destination Account) and all the items existing in the Source Account at the moment selected are recreated in the Destination Account, including the folder structure and all the user's data. All restored items will be created in the Current Primary Store unless the `Obey HSM Policy` box is checked. [ Add maybe the following : Which would then apply the HSM policies if they exist during the restore. Otherwise, HSM will be applied during your normal HSM schedule. ]”

WARNING: When restoring data on a new account, shared items consistency is not preserved. This is because the original share rules refer to the original account's ID, [ Account ID – meaning? Email addr, zimbraID, etc ] not to the restored one's.

Running a Restore on New Account via the Administration Zimlet

A Restore on New Account can be ran in two ways [ This is under the header for “administration zimlet” but the section only ‘shows’ ONE way. There’s one sub-section/header - “from the account list”. Is the other method from the CLI? If so, sub-header throwing things off with the reference to "zimlet" and the placement of this statement underneath it. ]

From the account list

Running the restore from the Accounts tab in the Zimbra Administration Console allows to operate on users currently existing on the server.

[ Replace above with : From the Accounts tab in the Zimbra Administration Console, one can do a restore on users tha currently exist on the server. ]

If you need to restore a deleted user, please proceed to the restore via the Administration Zimlet.

  • In the Left Pane of the Administration Console select Accounts to show up [ Drop "up" ] the Accounts List.
  • Browse the list and select the account to be restored (Source) by clicking on it.
  • On the top bar, press the wheel and then the `Restore ` button.
  • Select Restore on New Account as the Restore Mode and enter the name of the new account (Destination) in the text box. You can then choose whether to Hide in GAL the new account or not. When you're done choosing, press Next
  • Choose the restore date. Day/Month/Year can be selected via a minical, the hour via a drop-down menu, minute and second via two text boxes. Click Next.
  • Verify all your choice in the Operation Summary window. You can also add additional email addresses to be notified when the Restore operation is finished. Pleas Please notice that the Admin account and the User who started the Restore procedure are notified by default.

Click Finish to start the Restore.

Running a Restore on New Account via the CLI

To start a Restore on New Account via the CLI, the doRestoreOnNewAccount command is available:

[ Chat Buddy reference below. This is not mentioned in the admin console example. At the beginning, you stated ‘shares’ won’t be restored – one might wonder why buddies can and how that will work. Don’t buddies have to be ‘accepted’? ]

[ Date format issue below, `28/09/2012 10:15:10` . Ah, BIG DEAL here. For the USA, this will be huge – the change of the date format from Month/Day/Year to Day/Month/Year. Also note, zimbra [zmtools for backup/restore/redo] use Year/Month/Day format. ]

Syntax:
   zxsuite backup doRestoreOnNewAccount {source_account} {destination_account} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                       TYPE                  EXPECTED VALUES
source_account(M)          Account Name
destination_account(M)     Account Name/ID
date(M)                    Date                  `dd/MM/yyyy HH:mm:ss`|last
restore_chat_buddies(O)    Boolean               true|false
notifications(O)           Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup dorestoreonnewaccount John NewJohn `28/09/2012 10:15:10`
Restores John's account in a new account named NewJohn

Undelete Restore

What is Undelete Restore?

Undelete Restore is one of the Restore Modes available in Backup NG. [Are these sentences really necessary? They are repeated for each of the various restore modes.]

It allows an administrator to restore all items deleted [Should we mention what deleted means? Meaning a reference to the ‘dumpster’ option/feature and if it is or isn’t enabled.] from a mailbox in a period of time and and [double and] put them in a dedicated Zimbra folder inside the mailbox itself.

How does it work?

During an Undelete Restore the Backup NG engine searches the backup datastore for items flagged as DELETED and restores them in a dedicated folder in the selected Mailbox. [There’s no more context to this “dedicated folder” for undelete restore. There’s no ‘option’ to pick/create a target folder for the command and the doc doesn’t state what to look for or expect of this ‘dedicated folder’.]

[Moved warning to new paragraph]

WARNING: In order to allow to deal with IMAP-deleted emails in a more comfortable way for the user: the deleted IMAP flag will now be stripped from any restored item so that the item itself is visible in the Zimbra WebClient.

Running an Undelete Restore

Via the Administration Console

  • In the Left Pane of the Administration Console select Accounts to show up [Drop the up] the Accounts List.
  • Browse the list and select the account to be restored (Source) by clicking on it.
  • On the top bar, press the wheel and then the Restore button". [accidently included a " ?]
  • Select `Undelete` as the Restore Mode and press Next.
  • Choose the restore date-time slot. Day/Month/Year can be selected via a minical, the hour via a drop-down menu, minute and second via two text boxes. Click Next.
  • Verify all your choice in the Operation Summary window. You can also add additional email addresses to be notified when the Restore operation is finished. Pleas Please notice that the Admin account and the User who started the Restore procedure are notified by default.
  • Click Finish to start the Restore.

Via the CLI

To start an Undelete Restore operation, the `doUndelete` command is available:

Syntax:
   zxsuite backup doUndelete {account} {"dd/MM/yyyy HH:mm:ss"|first} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                TYPE                  EXPECTED VALUES
account(M)          Account Name
start_date(M)       Date                  `dd/MM/yyyy HH:mm:ss`|first
end_date(M)         Date                  `dd/MM/yyyy HH:mm:ss`|last
notifications(O)    Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doundelete John `08/10/2012 10:15:00` last
Performs an undelete on John's account of all items created between 08/10/2012 10:15:00 and the latest data available

External Restore

What is the External Restore?

I would make these subject lines be, "What is the [feature name] feature?"

The External Restore is one of the Restore Modes of Backup NG.

Doesn’t really answer the “what is” , does it?

How does it work?

The External Restore adds to the current Zimbra server all the data, metadata and configurations stored on an External Backup.

The workflow of the import procedure is the following:

PHASE1

  • Operation Started notification
  • Read Server Backup Data I would replace Server with External or Exported.
  • Create empty Domains What about the initial DOMAIN and COS created with a newly installed/setup ZCS server ? What if they exist in the External Backup?
  • Create needed COS (only those effectively used by the imported accounts)
  • Create empty DLs
  • Create empty Accounts Account without attributes first and then goes back doing all attributes to each account afterward? Or does it create an empty account , apply attributes to that account – continue to next account?
  • Restore all Accounts' attributes
  • Restore all Domains' attributes
  • Restore all DLs' attributes and share informations information
  • PHASE1 Feedback Notification

What about stating when the account is ‘initially operational’ and to what degree.

PHASE2

  • Restore all Items Meaning? “mail blobs”, calendar events, “IM/buddy stuff”, tasks, briefcase, etc?

PHASE3

  • Restore all Mountpoints and Datasources huh? mountpoint = shares? datasources= galsync accounts?
  • Operation Ended notification with complete feedback

Before you start

External Restore doesn't mention at all the status of the 'accounts' during this process? A common assumption, IMO, would be nothing is available until it's completed. We should make sure admin's know that isn't true.

If Backup NG is already initialized on the destination server, [Isn’t this a prereq anyways prior to the restore from running?] disable the RealTime Scanner in order to improve both memory usage and I/O performances.

In order to reduce the I/O overhead and the amount of disk space used for the migration, advanced users may tweak or disable Zimbra's RedoLog for the duration of the import. [As stated earlier, we should have more on the redolog and this could reference back to that section for details on ‘tweaking it’ if needed and ‘why’.]

In order to further reduce the amount of disk space used, it's possible to enable compression on your Current Primary Volume before starting the import. If you do not wish to use a compressed Primary Volume after migration, it's possible to create a new and uncompressed Primary Volume, set it to `Current` and the switch the old one to `Secondary`. All of this can be done using the HSM NG module. [Lacking any reference to HSM policy being applied or not during restore – if there’s an option or not, if it’s CLI only option, etc]

Running an External Restore

Via the Administration Zimlet

  • Click the Backup NG tab.
  • Click on the Import Backup button under Import/Export to open the Import Backup wizard.
  • Enter the Destination Path in the textbox and press Forward. The software will check if the destination folder contains a valid backup and whether the zimbra user has Read permissions or not.
  • Select the domains you want to import and press Forward. reference all option?
  • Select the accounts you want to import and press Forward. reference all option?
  • Verify all your choices in the Operation Summary window. You can also add additional email addresses to be notified when the Restore operation is finished. Please notice that the Admin account and the User who started the Restore procedure are notified by default. It should be more clear when we use this, it occurred earlier, to say “Global Admin or User Global Admin”. A ‘domain admin’ or ‘user’ can’t do restores.

Via the CLI

To start an External Restore operation, the doExternalRestore command is available:

Syntax:
   zxsuite backup doExternalRestore {source_path} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                          TYPE                 EXPECTED VALUES    DEFAULT
source_path(M)                Path
accounts(O)                   Account Name[,..]                       all
domains(O)                    Domain Name[,..]                        all
filter_deleted(O)             Boolean              true|false         true
skip_system_accounts(O)       Boolean              true|false         true
skip_aliases(O)               Boolean              true|false         false
skip_distribution_lists(O)    Boolean              true|false         false
provisioning_only(O)          Boolean              true|false         false
skip_coses(O)                 Boolean              true|false         false
notifications(O)              Email Address

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doexternalrestore /opt/zimbra/backup/restorePath/ accounts [email protected],[email protected] domains example.com filter_deleted false skip_system_accounts false
Restores the example.com domain, including all system accounts, and the [email protected] and [email protected] accounts from a backup located in /opt/zimbra/backup/restorePath/

Speeding up the restore through multithreading

The concurrent_accounts parameter allows to restore the restoring of multiple accounts at the same time, thus greatly speeding up the restore process. This feature is not available via the GUI. But it also doesn’t show up in the above help output for the CLI either!

WARNING: Albeit resource consumption does not grow linearily linearly with the number of accounts restored at the same time, it can easily become taxing - start from a low number of concurrent accounts, raising it according to your server's performances. [This implies it’s a variable you can tweak/adjust against a running restore process and not subsequent ones an admin might do [batch restores].]

Note again, concurrent_accounts doesn't show up in the help output from CLI above.

Usage example:

zxsuite backup doExternalRestore /tmp/external1 domains example0.com,example1.com concurrent_accounts 5

Restores the example0.com and example1.com domain, excluding system accounts, restoring 5 accounts at same time from a backup located in /tmp/external1

After the Restore: Message Deduplication

Running a volume-wide deduplication with HSM NG is highly suggested after an External Restore, since the native deduplication system might be ineffective when sequentially importing accounts.

Maybe include the CLI command with the help flag here?

Restore Deleted Account

What is the Restore Deleted Account?

The Restore Deleted Account procedure allows you to restore a the contents and preferences of a mailbox as it was when said mailbox was deleted, in a completely new account.

How does it work?

When a Restore Deleted Account starts a new account is created (the Destination Account) and all the items existing in the Source Account [Sentence structure is poor, needs to be rewritten] at the moment of the deletion are recreated in the Destination Account, including the folder structure and all the user's data. All restored items will be created in the Current Primary Store unless the `Obey HSM Policy` box is checked.

[Above, new account is created] Is the new account available as soon as it's created even though items are being restored? If so, this might be worth pointing out - especially in cases of a large mailbox. The "old method" would have the sys admin waiting for completion of the restore prior to telling the user they can login.

[Above, Obey HSM Policy] Maybe one more sentence to describe how this works if the option is enabled.

WARNING: When restoring data on a new account, shared items consistency is not preserved. This is because the original share rules refer to the original account's ID, not to the restored one's.

But, is it possible to get what the shares USE TO BE - even if it's in a text format. This could then be a guide for the user/admin to manually set them up using the new account id. If true, include in guide.

From the Backup NG tab

  • In the Left Pane of the Administration Console select Backup NG to show up [Drop the up] the Backup NG tab.
  • On the Top Bar push the Restore Deleted Account button.
  • Choose the restore date. Day/Month/Year can be selected via a minical, the hour via a drop-down menu, minute and second via two text boxes. Click `Next`.

Do I need a date? User will want to know they have option of 'latest' or something like that.

  • Browse the list and select the account to be restored (Source) by clicking on it.

What if I had more than one account I wanted to restore? zmrestore, I believe, could do this - at least from CLI.

  • Enter the name of the new account (Destination) in the text box. You can then choose whether to Hide in GAL the new account or not. When you're done choosing, press `Next`.

Above point about multiple user restore via this method - zmrestore had option called prefix , is this available in the Backup NG module? [what a NE user might ask]

  • Verify all your choice choices in the Operation Summary window. You can also add additional email addresses to be notified when the Restore operation is finished. Pleas Please notice that the Admin account and the User who started the Restore procedure are notified by default.

[Above, additional email addresses to be notified] Could you add in the 'new' email address as well? Note comment above, about if the account is usable as soon as it's created even though the restore still might be happening in the background. This would be nice for admin's since then they could just via the notification email let the user know the restore if completed - rather than a manual action.

  • Click Finish to start the Restore.

Item Restore

What is the Item Restore?

The Item Restore is one of the Restore Modes of Backup NG.

How does it work?

A single item is restored from the backup to the owner's account. Any type of item can be restored this way.

Running an Item Restore

Via the Administration Zimlet

The Item Restore is only available through the CLI

Via the CLI

To start an Item Restore operation, the `doItemRestore` command is available:

Syntax:
   zxsuite backup doItemRestore {account_name} {item_id} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                 TYPE
account_name(M)      Account Name
item_id(M)           Integer
restore_folder(O)    String

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doitemrestore [email protected] 4784
Restores item 4784 in the `[email protected]` mailbox

How to obtain the itemID

IMO - this should come before the example of doing the restore since it's your first step anyways.

The itemID is one of the so-called metadata of an item consisting in of an univoque A what code? code that identifies an item in a mailbox.

Along with all other metadata, it is stored in a file inside the items directory of the proper account in

[backup path]/accounts/[accountID]/items/[last 2 digits of itemID]/[itemID]`

e.g.:

Item 2057 of account 4a217bb3-6861-4c9f-80f8-f345ae2897b5, default backup path` +
/opt/zimbra/backup/ng/accounts/4a217bb3-6861-4c9f-80f8-f345ae2897b5/items/57/2057`

[below, plaintext file, so tool...] compressed or not compressed? zgrep vs grep Might clarify.

Metadata are stored in a plaintext file, so tools like grep and find can be used to search for contents. In order to see the metadata contained in a file in a more readable format you can use the zxsuite backup getItem command:

Syntax:
   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID
item(M)           Integer
backup_path(O)    Path                                          /opt/zimbra/backup/ng/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup getitem a7300a00-56ec-46c3-9773-c6ef7c4f3636 1
Shows item with id = 1 belonging to account a7300a00-56ec-46c3-9773-c6ef7c4f3636

[Above is source doc has help output listed twice.

[email protected]:~$ zxsuite backup getitem

command getItem requires more parameters

Syntax:
   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID
item(M)           Integer
backup_path(O)    Path                                          /opt/zimbra/backup/ng/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup getitem a7300a00-56ec-46c3-9773-c6ef7c4f3636 1
Shows item with id = 1 belonging to account a7300a00-56ec-46c3-9773-c6ef7c4f3636

Real Life example

Let's say a user moves one item to the Trash...

Wait, why an example of an item in trash and then deleted from trash? What about doUndelete ? Also, the doItemRestore should include doItemSearch example before the getItem one - IMO.

2013-07-18 15:22:01,495 INFO  [btpool0-4361://localhost/service/soap/MsgActionRequest [[email protected];mid=2538;oip=258.236.789.647;ua=zclient/7.2.4_GA_2900;] mailop - moving Message (id=339) to Folder Trash (id=3)

...and empties the trash.

2013-07-18 15:25:08,962 INFO  [btpool0-4364://localhost/service/soap/FolderActionRequest] [[email protected];mid=2538;oip=258.236.789.647;ua=zclient/7.2.4_GA_2900;] mailbox - Emptying 9 items from /Trash, removeSubfolders=true.

She then calls the Administrator in order to restore the deleted item. Knowing the itemID and the email address, the mighty Administrator only needs to run

zxsuite backup doItemRestore [email protected] 339

as the zimbra user to restore the missing item.

Disaster Recovery

As mentioned below about other components being absent from this section - would it be best to label as "Individual Mailstore Disaster Recovery or single ZCS Server Recovery"?

The disaster

Completely assumes a single server ZCS env. Doesn't even address that Backup NG is ONLY on mailstores

What can go wrong

To classify a problem under Disaster, one or more of the following must be of happened:

  • Hardware failure of one or more vital filesystems (such as / or /opt/zimbra/)
  • Contents of a vital filesystem made unusable by internal or external factors (like a careless *rm ** or an external intrusion)
  • Hardware failure of the physical machine hosting the Zimbra service or of the related virtualization infrastructure
  • A critical failure on a software or OS update/upgrade

Minimizing the chances

Some quick suggestions in order to minimize the chances of a disaster:

  • Always keep vital filesystems on different drives (namely /, /opt/zimbra/ and your Backup NG path)
  • Use a monitoring/alerting tool for your server to become aware of problems as soon as they appear
  • Carefully plan your updates and migrations

How about mentioning the best practice of actually DOING A DR walk-through test prior to attempting it for the first time when a REAL DR hits your production server?

The Recovery

How to recover your system

Doesn't explore/discuss/cover multi-ZCS server env. LDAP, MariaDB, etc....

The recovery of a system is divided in into 2 steps:

  • Base system recovery (OS installation and configuration, Zimbra installation and base configuration)
  • Data recovery (reimporting the last available data to the Zimbra server, including Domain and User Configurations, Classes of Services and mailbox contents)

How can Backup NG help with recovery

The Import Backup feature of Backup NG provides an easy and safe way to perform step 2 of a Recovery.

Using the old server's Backup Path as the import path allows you to restore a basic installation of Zimbra to the last valid moment of your old server.

The Recovery Process

  • Install Zimbra on a new server and configure the Server and Global settings.

You could install it again on the old server assuming you could maintain the old backup dir on it.

  • Install Network NG Modules on the new server.

If they are installing NE - the NG Modules are already installed. It's about configuring them rather than 'installing them'. Might point out if they should have them enabled/disabled/configured as they "were" prior to doing the restore or not. Of course, they'll know backup is needed but they might wonder about the other modules and if they also have to be configured as they were for the restore to work as expected.

  • Mount the Backup folder of the old server on the new one. If this is not available, you use the last External Backup available or the latest [rewrite to] "...,then use the most recent External Backup you might have available."
  • Begin an External Restore on the new server using the following CLI command:
zxsuite backup doExternalRestore /path/to/the/old/store

Why not show the help output for this and the various options for it? Why /store vs /backup?

  • The External Restore operation will immediatly immediately create the domains, accounts and distribution lists, so as soon as the first part of the Restore is completed (check your Network NG Modules Notifications) [what are they looking for? what will the message look like?] the system will be ready to be used by your users. Emails and other mailbox items [Might reference "shares" in this section somewhere also. When are they done or is it a manual post-restore action.] will be restored afterwards.

Settings and Configs

[section doesn't mention ldap or 'non-mailstore' services and their settings/configs but it does mention 'postfix' conf directory. Doesn't state - IF you have postfix/mta services on the mailstore or not. Doesn't indicate if it's pulling from mta's in env or ONLY if mta is installed on the mailstore.]

Server and Global settings are backed up but are not restored automatically: Backup NG's high-level integration with Zimbra allows you to restore your data to a server with a different OS/Zimbra Release/Networking/Storage setup without any constraints other than the minimum Zimbra version required to run Network NG Modules.

Whether you wish to create a perfect copy of the old server or just take a cue from the old server's settings to adapt those to a new environment, Backup NG comes with a very handy CLI command: getServerConfig.

[We shouldn't do this for help output, we should be using the help flag to show the output. They might think they can do this for all our commands and they might all 'expect' more parameters to run or not successful vs. dumping out help contents.]

[If a script is made to query the native config files, it would be extremely nice if it would grab relevant data just needed for an installation and order in it's output with the flow of the installation process. Add option to the script --for-installation . Next RFE would then be to expand the install.sh script to take in this output to automatically do the installation with those values.]

[email protected]:~$ zxsuite backup getServerConfig
command getServerConfig requires more parameters


Syntax:
   zxsuite backup getServerConfig {standard|customizations} [attr1 value1 [attr2 value2...


PARAMETER LIST


NAME              TYPE               EXPECTED VALUES                       DEFAULT
type(M)           Multiple choice    standard|customizations
date(O)           String             `dd/MM/yyyy HH:mm:ss`|"last"|"all"
backup_path(O)    Path                                                     /opt/zimbra/backup/ng/
file(O)           String             Path to backup file
query(O)          String             section/id/key
verbose(O)        String                                                   false
colors(O)         String                                                   false


(M) == mandatory parameter, (O) == optional parameter


Usage example:


zxsuite backup getserverconfig standard date last
 Display the latest backup data for Server and Global configuration.
zxsuite backup getserverconfig standard file /path/to/backup/file
 Display the contents of a backup file instead of the current server backup.
zxsuite backup getserverconfig standard date last query zimlets/com_zimbra_ymemoticons colors true verbose true
 Displays all settings for the com_zimbra_ymemoticons zimlet, using colored output and high verbosity.

Specifically,

zxsuite backup getServerConfig standard backup_path /your/backup/path/ date last query / | less

will display the latest backed up configurations.

You can change the query argument to display specific settings, e.g.

[email protected]:~$ zxsuite backup getServerConfig standard date last backup_path /opt/zimbra/backup/ng/ query serverConfig/zimbraMailMode/test.domain.com


config date_______________________________________________________________________________________________28/02/2014 04:01:14 CET
test.domain.com____________________________________________________________________________________________________________both
....

The \{zimbrahome}/conf/ and \{zimbrahome}/postfix/conf/ directories are
backed up aswell:

....
[email protected]:~$ zxsuite backup getServerConfig customizations date last verbose true
ATTENTION: These files contain the directories {zimbraHome}/conf/ and {zimbraHome}/postfix/conf/ compressed into a single archive.
           Restore can only be performed manually. Do it only if you know what you're doing.

        archives

                filename                                                    customizations_28_02_14#04_01_14.tar.gz
                path                                                        /opt/zimbra/backup/ng/server/
                modify date                                                 28/02/2014 04:01:14 CET

VMs and Snapshots

On "snapshots" see also zimbra's reference to snapshots : https://github.com/Zimbra/adminguide/blob/master/backuprestore.adoc#using-snapshots-to-backup-and-restore]

Thanks to the advent of highly evolved virtualization solutions in the past years, Virtual Machines are now the most common way to deploy server solutions such as Zimbra Collaboration Suite.

Most hypervisors feature customizable snapshot capabilites [capabilites vs capabilities], and snapshot-based VM backup systems: in case of a disaster, it's always possible to roll back to the latest snapshot and import the missing data using the `External Restore` feature of Backup NG - using the server's backup path as the import path. [Needs rewording. First sentence is confusing with the transition to the "in case of disaster"]

Disaster Recovery from a previous VM state

Snapshot-based backup systems allow you to keep a `frozen` copy of a VM in a valid state and rollback to it at will. To 100% ensure data consistency it's better to take snapshot copies of switched off VMs, but this is not mandatory.

When using this kinds of systems, it's vital to make sure that the Backup Path isn't either part of the snapshot (e.g. by setting the vdisk to Independent Persistend [persistend vs Persistent] in VMWare ESX/i) or altered in any way when rolling back in order for the missing data to be available for import.

[above] This advise really should be noted in the upgrade/install documents and the setting up/initialization of Backup NG. The last thing we want admin's having to do is 'initialize' twice - impacting their server's performance.

[above] Where does the zimbra redolog fit into this ?

In order to perform a Disaster Recovery from a previous machine state with Backup NG you need to:

  • Restore the last valid backup into a separate (clone) VM in an isolated network, making sure that users can't access it and that both incoming and outgoing emails are not delivered.
  • Switch on the clone and wait for Zimbra to start.
  • Disable Backup NG's RealTime Scanner.
  • Connect the Virtual Disk containing the untampered Backup Path to the clone and mount it (on a different path).
  • Start an External Restore using the Backup Path as the Import Path.

Doing so will parse all items in the Backup Path and import the missing ones, speeding up the Disaster Recovery by a good measure. This These steps can be repeated as many time as needed as long as user access and mail traffic is inhibited.

[last sentence] I can see someone reading this section and then this sentence and go... Let's setup a VM clone, give it a new ip address, new virt disk for the zimbra volume but then mount [readonly] the production hosts backup path. We'll then run this process throughout the day and we'll basically have a DR/Failover system. And if they read the incremental migration wiki - they'll wonder why it mentions warning about data under this section. URL for comment: https://wiki.zextras.com/wiki/ZxBackup:_Incremental_migration_with_ZeXtras_Backup#What_will_NOT_be_migrated.

After the restore is completed, make sure that everything is functional and restore user access and mail traffic.

The Aftermath

What now?

Just initialize a new Backup Path and store the old one until you please - should you need to restore any content from before the disaster.

Unrestorable items

How can I check if all of my items have been restored?

It's very easy: just check the appropriate Operation Completed notification you received as soon as the restore operation has been completed. It can be viewed in the `Notifications` section of the Administration Zimlet and it's also emailed to the address you specified in the Core section of the Administration Zimlet as the Notification E-Mail recipient address.

The `skipped items` section contains a per-account list of unrestored items:

  [...]
  - stats -
  Restored Items: 15233
  Skipped Items:  125
  Unrestored Items: 10

  - unrestored items -
  account: [email protected]
  unrestored items: 1255,1369

  account: [email protected]
  unrestored items: 49965

  account: [email protected]
  unrestored items: 856,13339,45200, 45655
  [...]

Skipped Items vs. Unrestored Items

Important distinction for the zimbra support team to be aware of.

  • Skipped item: an item that has been already restored, either during the current restore or in a previous one.
  • Unrestored item: an item that has not been restored due to an issue in the restore process.

Why some of my items have not been restored?

There are different possible causes, the most common of which are:

  • Read Error: Either the raw item or the metadata file is not readable due to an I/O exception or a permission issue.
  • Broken item: Both the the raw item or the metadata file are readable by Backup NG but their content is broken/corrupted.
  • Invalid item: Both the the raw item or the metadata file are readable and the content is correct, but Zimbra refuses to inject the item.

How can I identify unrestored items?

There are two ways to do so: via the CLI and via the Zimbra Webclient. The first way can be used to search for the item within the backup/import path while the second way can be used to view the items in the source server.

Identifying Unrestorable items through the CLI

The getItem command of the CLI can display an item and the related metadata extracting all informations from a backup path/external backup.

The syntax of the command is the following:

   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID
item(M)           Integer
backup_path(O)    Path                                          /opt/zimbra/backup/ng/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Thus, to extract the raw data and metadata informations of the item whose itemID is _49965_ belonging to [email protected]_ also including the full dump of the item's BLOB the command would be:

zxsuite backup getItem [email protected] 49965 dump_blob true

Identifying Unrestorable items through the Zimbra WebClient

The comma separated list of unrestored items displayed in the Operation Complete notification can be used as a search argument in the Zimbra Webclient to perform an item search.

To do so:

  • Log into the Zimbra Administration Console in the source server.
  • Use the `View Mail` feature to access the account in which the unrestored items are.
  • In the search box, enter *item:* followed by the comma separated list of itemIDs.
e.g.`
item: 856,13339,45200,45655`
WARNING: Remember that any search is executed only within the tab it is executed, so if you are running the search from the Email tab and get no results try to run the same search in the Address Book, Calendar,Tasks and Briefcase` tabs.

How can I restore unrestored items?

While an item not being restored is a clear sign of an issue, either with the item itself or with your current Zimbra setup, in some cases there are good chances of being able to restore an item even if it was not restored on the first import try.

In the following paragraphs you will find a collections of tips and tricks that can be helpful when dealing with different kinds of unrestorable items.

Items not restored because of a Read Error

A dutiful distinction must be done about the read errors that can cause items not to be restored:

  • hard errors: hardware failures and all other `destructive` errors that cause an unrecoverable data loss.
  • soft errors: `non-destructive` errors such as wrong permissions, filesystem errors, RAID issues (e.g.: broken RAID1 mirroring) etcetera.

While there is nothing much to do about hard errors, you can prevent or mitigate soft errors following these guidelines:

  • Run a filesystem check.
  • If using a RAID disk setup, check the array for possible issues (depending on RAID level).
  • Make sure that the 'zimbra' user has r/w access to the backup/import path, all its subfolders and all thereby contained files.
  • Carefully check the link quality of network-shared filesystems. If link quality is poor consider transferring the data with rsync.
  • If using SSHfs to remotely mount the backup/import path make sure to run the mount command as root using the `-o allow_other` option.

Items not restored because identified as Broken Items

Unfortunately, this is the worst category of unrestored items in terms of `salvageability`.

Based on the degree of corruption of the item, it might be possible to recover either a previous state or the raw object (this is only valid for emails). To identify the degree of corruption, the `getItem` CLI command comes handy:

   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID
item(M)           Integer
backup_path(O)    Path                                          /opt/zimbra/backup/ng/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Searching for the broken item setting the `backup_path` parameter to the import path and the `date` parameter to `all` will display all valid states for the item.

[email protected]:~$ zxsuite backup getItem [email protected] 24700 backup_path /mnt/import/ date all
       itemStates                              
               start_date                                                  12/07/2013 16:35:44
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
               start_date                                                  12/07/2013 17:04:33
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
               start_date                                                  15/07/2013 10:03:26
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

If the item is an email you will be able to recover a standard .eml file through the following steps:

  • Identify the latest valid state
/mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
              start_date                                                  15/07/2013 10:03:26
              type                                                        message
              deleted                                                     true
              blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
  • Identify the blob path
`blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=`
  • Use gzip to uncompress the BLOB file into an .eml file
[email protected]:~$ gunzip -c /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= > /tmp/restored.eml

[email protected]:~$ cat /tmp/restored.eml

Return-Path: [email protected]

Received: from test.example.com (LHLO test.example.com) (192.168.1.123)
by test.example.com with LMTP; Fri, 12 Jul 2013 16:35:43 +0200 (CEST)

Received: by test.example.com (Postfix, from userid 1001) id 4F34A120CC4; 
Fri, 12 Jul 2013 16:35:43 +0200 (CEST)
To: [email protected]
From: [email protected]
Subject: Service mailboxd started on test.example.com
Message-Id: <[email protected]>
Date: Fri, 12 Jul 2013 16:35:43 +0200 (CEST)

Jul 12 16:35:42 test zmconfigd[14198]: Service status change: test.example.com mailboxd changed from stopped to running
  • Done! You can now import the .eml file into the appropriate mailbox using your favorite client.

Items not restored because identified as Invalid Items

An item is identified as Invalid when, albeit being formally correct is discarded by Zimbra's LMTP Validator upon injection. This is common when importing items created on an older version of Zimbra to a newer one, as the validation rules are updated very often and because of this not all messages considered valid by a certain Zimbra version are still considered valid by a newer version.

If you experienced a lot of unrestored items during an import it might be a good idea to momentarily disable the LMTP validator and repeat the import:

  • To disable Zimbra's LMTP Validator run the following command as the Zimbra user:
zmlocalconfig -e zimbra_lmtp_validate_messages=false
  • Once the import is completed you can enable the LMTP validator running
zmlocalconfig -e zimbra_lmtp_validate_messages=true
WARNING: This is a dirty workaround, as items deemed invalid by the LMTP validator might cause display or mobile synchronization errors. Use at your own risk.

doCoherencyCheck

What is the Coherency Check?

The Coherency Check is a feature added in Network NG Modules 1.10.2 which performs a deeper check of a Backup Path than the one done by the SmartScan.

While the SmartScan works incrementally, by only checking items which have been modified since the last SmartScan, the Coherency Check performs a throughout check of all metadata and BLOBs in the backup path. What about the HSM volume/blob checking tools? How are they different, etc ?

It's specifically designed to detect corrupted metadata and BLOBs.

How does it work?

The Coherency Check verifies the integrity of every metadata in the backup path and of the related BLOBs: should any errors be found, running the check with the `fixBackup` option will move any orphaned or corrupted metadata/BLOB to a dedicated directory within the backup path.

When should a Coherency Check be executed?

  • At interval periods in order to make sure that everything is ok (e.g. every 3 or 6 months).
  • After a system crash.
  • After the filesystem or storage device containing the backup path experiences any issue.

Should the SmartScan detect a possible item corruption, a Coherency Check will be started automatically.

Hm.. probably should be explained or clarified more. If you have an option for 'smartscan' to start on ZCS start-up and it detects ONE possible item being corrupted it will then throw my server into a high i/o consuming activity [coherency check]? Would the admin be notified [email] of this? Option to cancel/kill the operation? Check the smartscan 'logs' to see if it's warranted or not? Does the coherency check stop the smart scan in this situation?

WARNING: The Coherency Check is highly I/O consuming, so make sure to run it only during off-peak periods.

Running a Coherency Check

Starting the Check via the Administration Zimlet

The Coherency Check is not available on the Administration Zimlet.

Starting the Check via the CLI

To start a Coherency Check via the CLI the `doCoherencyCheck` command is available

Syntax:
   zxsuite backup doCoherencyCheck {backup_path} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                TYPE                    EXPECTED VALUES    DEFAULT
backup_path(M)      Path
accounts(O)         Account Name/ID[,..]                       all
checkZimbra(O)      Boolean                 true|false         false
fixBackup(O)        Boolean                 true|false         false
notifications(O)    Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup docoherencycheck /opt/zimbra/backup/ng/ accounts [email protected],[email protected]
Performs a coherency check on /opt/zimbra/backup/ng/ for Jack's and John's accounts
zxsuite backup docoherencycheck /opt/zimbra/backup/ng/ fixBackup true
Performs a coherency check on /opt/zimbra/backup/ng/ and moves corrupted backup files and blob files not referenced by any metadata out of backup

Checking the status of a running check

To check the status of a running scan via the CLI the `monitor` command is available

Syntax:
   zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                 TYPE
operation_uuid(M)    Uiid
operation_host(O)    String

(M) == mandatory parameter, (O) == optional parameter

Taking Additional and Offsite backups of Backup NG's Datastore

Who watches the watchmen?

While mailserver backups were never really a concern for Juvenal or Plato, the concept of who watches the watchmen can also be applied to this field.

Having a [insert s] backup system is a great safety measure against data loss, but each backup system must be part of a broader backup strategy to ensure the highest possible level of reliability. The lack of a proper backup strategy gives a false sense of security, while actually turining turning even the best backup systems in the world into yet another breaking point.

Devising a backup strategy is no easy matter, and at some point , [insert comma] you will most likely be confronted with the following question: What if I lose the data I backed up?. The chances of this happening ultimately only depend on how you make and manage your backups: it's you're more likely to lose all of your backed up data if you store both your data and your backups in a single SATAII disk hard-drive than if you store your backed up data on a dedicated SAN using a RAID 1+0 setup.

In this article , [insert comma] you can find some suggestions and best practices in order to improve your backup strategy by making a backup of the Backup NG's Datastore and storing it offsite.

Making an additional backup of Backup NG's datastore

  • Atomicity: any transaction is committed and written to the disk only when completed.
  • Consistency: any commited committed transaction is valid and no invalid transaction will be committed and written to the disk.
  • Isolation: all transactions are executed sequentially so that no more than 1 transaction can affect the same item at once.
  • Durability: once a transaction is committed, it will stay so even in case of a crash (e.g. power loss or hardware failure).

The item above are NOT actually stated as features of Backup NG in this section. There is no real feature listing, highlights, table in the doc.

Due to this, it's very easy to make a backup of it. The best (and easiest) way to do so is by using *http://rsync.samba.org/[rsync]*. Specific options and parameters depend on many factors, such as the amount of data to be synced and the storage in use, while connecting to an rsync daemon instead of using a remote shell as a transport as it's usually much faster in transferring the data.

You won't need to stop neither [unncessary word] Zimbra nor the Real Time Scanner in order to make an additional backup of Backup NG's datastore using rsync, and you will be always able to stop the sync at any time and reprise it afterwards if needed.

Storing your Backup NG's datastore backup offsite

As seen in the previous paragraph, making a backup of Backup NG's Datastore is very easy, and the use of rsync makes it just as easy to store your backup in a remote location.

In order to optimize your backup strategy when dealing to with this kind of setup, the following best practices might help a lot:

  • If you schedule your rsync backups, make sure that you leave enough time between an rsync instance and the next one in order for the transfer to be completed.
  • Use the --delete options so that files that have been deleted in the source server are deleted in the destination one to avoid inconsistencies.
    • If you notice that using the `--delete` option takes too much time, schedule two different rsync instances: one with the `--delete` to be ran after the weekly Purge and one without such option.
  • Make sure you transfer the whole folder tree recursively starting from Backup NG's Backup Path. This includes Server Config backups and mapfiles.
  • Make sure the destination filesystem is case sensitive (just as Backup NG's Backup Path must be).
  • If you plan to restore directly from the remote location, make sure that the _zimbra_ user on your server has read and write permissions on the transferred data.
  • Expect to experience slowlynesses if your transfer speed is much higher than your storage throughtput (or vice versa). rewrite this or just remove it

Additional/Offsite backup F.A.Q.

Why shouldn't I use the Export Backup feature of Backup NG instead of rsync?

For many reasons:

  • The `Export Backup` feature is designed to perform migrations. It exports a `snapshot` which is an end in itself and was not designed to be managed incrementally, meaning that each time an Export Backup is ran

it'll probably take just as much time as the previous one, while using rsync is much more time-efficient.

  • Being a Backup NG operation, any other operation started while the Export Backup is running will be queued until the Export Backup is completed.

As mentioned before, we might want to have a dedicated section at the beginning of the backup content about the queue . Support will need to know this for support/troubleshooting.

  • An `Export Backup` operation has a higher impact on system resources than an rsync.
  • Should you need to stop an Export Backup operation, you won't be able to reprise it and you'll need to start from scratch.

Can I use this for Disaster Recovery?

Yes. Obviously, if your Backup Path is still available it's better to use that, as it will restore all items and settings to the last valid state, but should your Backup Path be lost you'll be able to use your additional/offsite backup. See the Something missing here from source

Can I use this to restore data on the server the backup copy belongs to?

Yes, but not through the `External Restore` operation since item and folder IDs are the same.

Hmm.. this is the first time this clarification/restriction has come up. Seems like it should of been mentioned in other sections and maybe explained more fully under the 'external restore' section.

The most appropriate steps to restore data from a copy of the backup path to the very same server are the following:

  • Stop the RealTime Scanner
  • Change the Backup Path to the copy you wish to restore your data from
  • Run either a `Restore on New Account` or a `Restore Deleted Account`

I would ask.. why not just have an option to restore "over existing" or "repair existing" .. seems like the limitation of those two options is pretty restrictive and might not give me what I really what for a solution in this circumstance. I mean, I would also think that in the end I would be better just standing up a new server and then doing the 'external restore' . "restore on new account' would mean shares are 'broken'.

  • Once the restore is over, change the Backup Path to the original one
  • Start the RealTime Scanner. A SmartScan will trigger to update the backup data

Can I use this to create an Active/Standby infrastructure?

No, because the `External Restore` operation does not perform any deletions. In the long run running several External Restores you'll end up filling up your mailboxes with unwanted stuff, since items deleted from the original mailbox will not be deleted on the `standby` server.

Again, under the vm snapshot section when it talks about running the restore multiple times it doesn't warn of this. Implies there isn't an issue at all when rerunning the restore.

The `External Restore` operation has been designed so that accounts will be available for use as soon as the operation is started, so that your users will be able to send and receive emails even if the restore is running.

Are there any other ways to do an Additional/Offsite backup of my system?

either expand upon or remove this Q&A

There are for sure, and some of them might even be better than the one described in this page. This are just guidelines that apply to the vast majority of the cases.

Multistore Informations

Backup NG in a Multistore Environment

Command execution in a Multistore Environment

The new Network NG Modules Administration Zimlet makes the management of multiple servers very easy. You can select a server from the Backup NG tab and perform all backup operations on that server, even if you are logged into the Zimbra Administration Console of another server. See the standard Something missing here from source

This also applies to the Something missing here from source

Specific differences between Singlestore and Multistore environments are: These aren't really differences - they are things simply to be aware of when in a multi-server env.

  • In a Multistore Environment, Restore on New Account operations ALWAYS create the new account in the Source account's mailbox server.
  • All operations are logged on the target server, not in the server that launched the operation.
  • If a wrong target server for an operation is chosen, Zimbra automatically proxies the operation request to the right server.

Backup and Restore

Backup and Restore in a Multistore environment will work exactly like in a Singlestore environment.

The different servers will be configured and managed separately via the Adminsitration Administration Zimlet, but certain operations like Live Full Scan and Stop All Operations can be 'broadcast' to all the mailstores via the zxsuite CLI using the --hostname all_servers option. This applies also to Backup NG settings (see the CLI wiki page for more details).

[{above} On --hostnane all_servers] Might want to make more clear that this is a variable to the base zxsuite vs. the module variable. zxsuite [--host] [--json] {module} {action} [options] Options (for zxsuite - see each command's single help for command options): --host [hostname|ip] - Specify a target host for the command. Leave blank for localhost. Use all_servers to broadcast the command to all Zimbra servers (with ZxSuite installed.) This all_servers only shows up there in the help output. URL for comment http://www.digitalhandshakes.com/wiki/index.php/Ajcody-zextras-zxsuite-NG-help#zxsuite_help_backup_doItemSearch

[{above} last sentence] What exact applies also to the Backup NG settings? This isn't clear

Backup and Restore operations are managed as follows:

  • Smartscans can be executed on single servers via the Administration Zimlet or on multiple servers via the CLI.
  • Restores can be started from the `Accounts` tab in the Zimbra Admin Console, from each server tab in the Backup NG menu of the Administration Zimlet and via the CLI. The differences between these methods are:
Operation started from: Options
Accounts tab The selected account's restore is automatically started in the proper server.
Server tab Any accounts eligible for a restore on the selected server can be chosen as the restore source
CLI Any account on any server can restored, but there is no automatic server selection.

Export and Import

Export and Import functions are those which differ the most when performed on a Multistore environment.

Here are the basic scenarios:

Export from a Singlestore and import to a Multistore

Importing multiple accounts of a single domain to different store will break the consistency of ALL the items that are shared from/to a mailbox on a different server.

A command in the CLI is available in order to fix the shares for accounts imported on differen servers.

Export from a Multistore and import to a Single or Multistore

Two different Scenarios apply here:

  • `Mirror` import: Same number of source and destination mailstores, each export is imported on a different server. This will break the consistency of ALL the items that are shared from/to a mailbox on a different server. The `doCheckShares` and `doFixShares` CLI commands are available to check and fix share consistency (see below).
  • `Composite` import: Same or different number of source and destination servers. Domains or accounts are manually imported into different servers. This will break the consistency of ALL the items that are shared from/to a mailbox on a different server. The `doCheckShares` and `doFixShares` CLI commands are available to check and fix share consistency (see below)

The `doCheckShares` and `doFixShares` commands

The `doCheckShares` command will parse all share informations in local accounts and report any error:

[email protected]:~$ zxsuite help backup doCheckShares

Syntax:
   zxsuite backup doCheckShares

Usage example:

zxsuite backup doCheckShares
Check all shares on local accounts

The `doFixShares` will fix all share inconsistencies using a migration

[email protected]:~$ zxsuite help backup doFixShares

Syntax:
   zxsuite backup doFixShares {import_idmap_file}


PARAMETER LIST

NAME                    TYPE
import_idmap_file(M)    String

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doFixShares idmap_file
Fixes the shares' consistency after an import according to the mapping contained in the /opt/zimbra/backup/ng/idmap_file

Operation queue and queue management

Backup NG's Operation Queue

Every time a Backup NG operation is started - either manually or through scheduling - it is enqueued in a dedicated unprioritized FIFO queue. Each operation is executed as soon as any preceding operation is dequeued (either because it has been completed or terminated).

The queue system affects the following operations:

  • External Backup
  • All restore operations
  • Smartscan

Changes to Backup NG's configuration are not enqueued and are applied immediately.

Operation Queue Management

Through the Administration Console

Viewing the Queue

In order to view the operation queue, access the `Notifications` tab in the Administration Zimlet and click the `Operation Queue` button.

WARNING: The Administration Zimlet displays operations enqueued both by the Backup NG and the HSM NG modules in a single view. This is just a design choice, as the two queues are completely separated, meaning that one Backup NG operation and one HSM NG operation can be running at the same time.
Emptying the queue

In order to stop the current operation and empty Backup NG's operation queue, enter the `Backup NG` tab in the Administration Zimlet and click the `Stop all Operations` button.

Through the CLI

Viewing the Queue

In order to view Backup NG's operation queue, the `getAllOperations` command is available:

[email protected]:~$ zxsuite help backup getAllOperations

Syntax:
   zxsuite backup getAllOperations [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME          TYPE       EXPECTED VALUES    DEFAULT
verbose(O)    Boolean    true|false         false

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup getAllOperations
Shows all running and queued operations
Emptying the queue

In order to stop the current operation and empty Backup NG's operation queue, the `doStopAllOperations` command is available:

[email protected]:~$ zxsuite help backup doStopAllOperations

Syntax:
   zxsuite backup doStopAllOperations


Usage example:

zxsuite backup doStopAllOperations
Stops all running operations
Removing a single operation from the queue

In order to stop the current operation or to remove a specific operation from the queue, the `doStopOperation` command is available:

[email protected]:~$ zxsuite help backup doStopOperation

Syntax:
   zxsuite backup doStopOperation {operation_uuid}


PARAMETER LIST

NAME                 TYPE
operation_uuid(M)    Uiid

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doStopOperation 30ed9eb9-eb28-4ca6-b65e-9940654b8601
Stops operation with id = 30ed9eb9-eb28-4ca6-b65e-9940654b8601

COS-level Backup Management

What is COS-level Backup Management

COS-level Backup Management allows the administrator to disable ALL Backup NG functions for a whole Class of Service in order lower storage usage.

How does COS-level Backup Management work?

What happens if I disable the Backup NG Module for a Class of Service?

  • The Real Time Scanner will ignore all accounts in such COS.
  • The Export Backup function WILL NOT EXPORT accounts in such COS.
  • Accounts in such COS will be treated as `Deleted` by the backup system. This means that after the Data Retention period expires all data for such accounts will be purged from the backup store. Re-enabling the backup for a Class of Service will reset this.

How is the backup enabled/backup disabled information saved?

Disabling the backup for a Class of Service will add the following marker to the Class of Service's Notes field: ${ZxBackup_Disabled}

While the Notes field remains fully editable and usable, changing or deleting this marker will re-enable the backup for the COS.

Incremental migration with Backup NG

After doing a review of this section, at least for the wiki site - I would strongly recommend having two documents. One where the import is the 'full' method. The other document, where it uses the provisioning only mode first with the data being imported afterward [after the switch over]. I just see there being too many possible fubar situations happening unless the methods are clearly distinct.

Lacks clarity on multi-server ZCS environment. What if you have a single server and are going to a new multi-server ZCS environment? Does that break shares in some cases? And so forth.

Description

  • This guide describes how to perform an Incremental Migration using Backup NG.
  • It's specifically designed for the migration of a production environment, minimizing the downtime and aiming to be transparent for the users.
  • If correctly planned and executed, your mail system won't suffer any downtime and the impact on the users will be close to zero.
  • [Migration adoc has also] The source server of the migration requires either Zextras Suite or Zimbra Suite Plus.
    • [Migration adoc has also] This guide applies for migrations from any Zimbra version supported by either of those to Zimbra 8.8.
  • All the CLI commands in this guide must be executed as the Zimbra user unless otherwise specified.

What will be migrated

  • Emails and Email Folders.
  • Contacts and Address Books.
  • Appontments Appointments [misspelt in migratino adoc also] and Calendars.
  • Tasks and Tasklists Task Lists [change in migration adoc also].
  • Files and Briefcases.
  • Share informations.
  • User Preferences.
  • User Settings.
  • Class of Service Settings.
  • Domain Settings.

What will NOT be migrated

  • Server settings (migrated for reference but not restored).
  • Global Settings (migrated for reference but not restored).
  • Customizations (Postfix, Jetty, etc...).
  • Items moved or deleted during the process will not be moved or deleted on the destination server.
  • Preferences (e.g. passwords) changed during the process will be reset upon each import.
WARNING: The Incremental Migration is not designed to set up a server-to-server mirroring. Using multiple imports to create a mirrored copy of the source server won't create a *mirrored* copy at all, since no deletions are performed by the Import process.

Source Data

NOTE This section is in the migration adoc BUT NOT the admin guide.

Data from the source server is obtained through either Zextras Suite or Zimbra Suite Plus.

This isn't clear. Be specific. "...by way of running a backup export with the backup module.

Should we mention ZCS supported versions here or in the next section?

Software installation

NOTE This section is in the migration adoc BUT NOT the admin guide.

Once you obtained the package of either Zextras Suite or Zimbra Suite Plus, the installation process is the following:

  • Copy the package over to your server, in a directory owned by the root user.
  • Unpack the package using tar zxf.
  • Enter the newly created directory called either zextras_suite-[version] or zimbra_suite_plus-[version].
  • As root, run the installation script ./install.sh all.
  • Follow the installation wizard, accepting the software EULA and installing both the Core and the Zimlet component.
  • Once the installation is completed, you can proceed with the guide.
WARNING: Installing Zimbra Suite Plus or Zextras Suite requires a mailboxd service restart, which will be performed during the installation.

Pre-migration checks

Servers

  • The source server: any Zimbra server can be the source of your migration, provided that it's running Backup NG or Zimbra Suite Plus.
  • The destination server: any Zimbra server can be the destination of your migratio migration [Note - also wrong in migration adoc], provided that it's running Backup NG.

Storage

  • On the Source server: If Backup NG is not currently enabled on the source server, make sure you have an amount of free disk space comparable to the size of the /opt/zimbra/store/ folder (the exported data is compressed through the gzip algorythm algorithm [Note - also wrong in migration adoc], and all zimbra items are deduplicated, usually reducing the size of exported to the 70% of the original size).
  • On the Destination server: Make sure you have an amount of free space greater than the size of the /opt/zimbra/store/ and of the export folders on the source server combined.

Wait, unless we mention the import will NOT be doing dedup's on recovery and note the more intensive/time-consuming 'apply hsm on import/restore' option, they'll need better guidance on sizing their partition/volumes on the new server. Would need to also briefly note abou the Deduplication step that can be done post-restore.

Data Transfer

While you can choose to transfer the data in any other way, rsync is our method of choice as it's a good compromise between speed and convenience.

The main data transfer is executed while the source server is still active and functional. However, since the transfer is performed via the [insert the] network, carefully plan your transfer in advance so that you'll have transfered transferred [Also wrong in migration adoc] all of your data before migrating.

Alternative Ways to transfer your data

Anything spanning from remote mount to physical move of the drive is ok as long as it suits your needs.

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
--Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6.

DNS

Set the TTL value of your MX record to 300 on your `real` DNS. This will allow a fast switch between source and destination server.

The Setup

Step 1: Coherency checks

In order to avoid any possible data-related issue, run the following checks on the source server:

  • zmblobchk: this command checks the consistency between Zimbra's metadata and BLOBs.

Why not recommend zxsuite hsm doCheckBlobs ?

Repair any error found as described in Zimbra's official documentation.

Running a reindex of all mailboxes is also suggested.

No example or reference. Doing it for all - how? zmprov doesn't have an option for all.

Step 2: Network NG Modules setup

Disable the Real Time Scanner on both servers:

zxsuite backup setProperty ZxBackup_RealTimeScanner false
WARNING: A dedicated device for the data export is strongly recommended in order to improve the export performance and to lower the impact on the performances of the running system.

Such device must be mounted on the `/opt/zimbra/backup/` path and the Zimbra user must have r/w permissions on it

It must be mounted on /opt/zimbra/backup ?

Step 3: Data Export (SmartScan)

Why is this section Data Export and SmartScan? They should be two different sections. SmartScan and then Data Export. SmartScan has nothing to do with Export, IMO - it confuses the reader.

In SmartScan section, explain what it is and why we are doing it prior to the export.

Run a SmartScan on the source server:

zxsuite backup doSmartScan

Wait, there's something missing here. The actual export is missing! Also missing in migration adoc

This sentence should be moved into the new section I recommended above "data export" and out of the "smartscan" one.

All your data will be exported to the default backup path (/opt/zimbra/backup/ng/).

No it won't - not like the sentence implies. doExport has a variable for path to export data to.

Pro-Tip: Single Domains Export

You can also choose to only migrate one or more domains instead of all of them. To do so, run the following command instead of the SmartScan:

zxsuite backup doExport /path/to/export/folder/ domains yourdomain.com,yourdomain2.com[..]

Mind that if you start with the SmartScan method you'll have to carry on the migration with such method, and if you start with the Single Domains method you'll have to carry on the migration with this one. The two methods cannot be mixed.

Missing a section for Exporting via the CLI . An example was given above in regards to the domain example but there was no example for the full server export. Also, should we include the doExport help output?

Data export (SmartScan) via the Administration Zimlet

You can also choose to export your data using the Administration Zimlet following

Step 4: Data Synchronization

WARNING: When you move the exported data to the destination server make sure that the destination folder is not Backup NG's backup path on the destination server in order to avoid any nuisiances nuisances [Also wrong in migration adoc] if you already use Backup NG or plan to do so on the destination server.

(You can skip this step if you choose to transfer your data by other means than rsync.)

Using rsync, copy the data contained in the /opt/zimbra/backup/ng/ on a directory in the destination server (make sure the Zimbra user has r/w permissions on such folder directory [Wrong in migtation adoc also] ). Use a terminal multiplexer like screen or tmux, this process command might need A LOT of time depending on network speed and amount of data involved.

[run this command as Root]
rsync -avH /opt/zimbra/backup/ng/ [email protected]:/path/for/the/data/
Alternate synchronization method

While the suggested method is great for high-bandwidth situations, the first synchronization can involve a lot of data. If you feel that the rsync method is too slow, you might consider a physical move of the device (or the proper disk file if running on a virtual environment).

After moving the disk, you can remotely mount it back to the source server (e.g. via SSHFS), as the additional synchronizations needed for the migration will involve much less data. In this case, be sure to remount the device on the source server as /opt/zimbra/backup/ng/ with all due the correct [fix in migration adoc also permissions.

Step 5: First import

Recommended changes here/below will need to be done in migration adoc also

Import all exported data to the destination server:

zxsuite backup doExternalRestore /path/for/the/data/

Now sit back and relax while Network NG Modules imports your data on the destination server.

Warning: Do not edit nor delete the This warning is missing text. Also wrong in migration adoc

Should we mention at the start of the import section about System Accounts? Why or why not to import them? What about ham/spam accts not being imported?

First import via the Administration Zimlet

You can also choose to import your data using the Administration Zimlet following While importing via the Administration Zimlet [< is that suppose to be quoted/linked? Indicating it's a section title] be sure to remove all System Accounts (like GalSync, Ham, Spam, Quarantine etc.) from the imported account list. Again, why remove them? What's the impact of losing the 'old system' accounts?

Step 5 (alternate): First import for large migrations [ADVANCED users only]

If you are to migrate a very large infrastructure where an export/import lasts for hours or even days, there is an alternative way to handle the migration from this point forward.

Instead of importing all of your data to the destination server, you can run a Provisioning Only import that will only create Domains, Classes of Service and Accounts on the destination server, skipping all mailbox contents.

zxsuite backup doExternalRestore /path/for/the/data/ provisioning_only TRUE

After doing this, switch the mailflow mail flow to the new server and, when the switch is completed, start the real import.

Wait, mail flow only or mail flow, DNS, logins, etc? Might indicate more details on how they'll proceed with the rest of the document and where it differs from the other method.

Also, I would replace the use of the word mail flow - it's too vague AND it lacks details on what steps it might mean. Fails to mention proxy, mta, ldap, and possible third party email client impact. DNS / zmhostname stuff will also probably have to be addressed. The section about getting server configs also fails to point out things like proxy ports etc. Things that MIGHT BE REQUIRED to be configured on the new server for a proper switch to occurr. Also fails to metnion certificats - commercial and self-signed.

zxsuite backup doExternalRestore /path/for/the/data/

This way, your users will now connect to the new server where new emails will be delivered while old emails are being restored.

This approach has it's its pros and cons, namely:

Pros
  • Since items are only imported once and never modified or deleted afterwards, using this method will result in less discrepancies than the `standard` incremental migration.
  • This is the option that has less impact on the source server (e.g. good if you are in a hurry to decommission it).
Cons
  • Depending on the timing of the operation, this method has a higher impact on your users due to the fact that items are restored WHILE they work on their mailbox.
  • Since the import is done on a running system, you might notice some slowdowns.

Wait!!! There is no logging reference/reporting for the import section. A sys-admin will like to know more about monitoring the import activity.

This section could be expanded to include more options/tuning or addressing basic questions like - "Can I make the CEO the first one restored?" , "Can I use this method on a per domain basis?" etc

The situation so far

Right now the vast majority of the data has already been imported to the destination server. The source server is still active and functional, and you are ready to perform the actual migration.

Unless you followed, "Step 5 (alternate): First import for large migrations [ADVANCED users only]" that is.

Actually, I'd either recommend rewriting this section or getting rid of it completely. Seems like there are two points to be made here 1. If you did the full import, you'll wait until it's completed before proceeding onto the next steps. 2. If you did the provision only method, you would proceed onto the steps and have to note at some point to do the full restore after the switch over has occurred.

The Migration

Step 6: Pre-migration checks

Before switching the mail flow, ALWAYS make sure that the new server is ready to become active (check your firewall, your DNS settings, your security systems etc.)

Step 7: The Switch

This needs clarity on the full import first method vs. the advance provision only method. Should be a big warning actually.

This is it, the migration moment has come! At the end of this step the destination server will be active and functional.

  • Repeat step 3, step 4 and step 5 (only new data will be exported and synchronized)

Maybe another reminder, if using the full import vs provision only method, that deletes will not be synced?

  • Switch the mail flow to the new server.
  • Once NO MORE EMAILS arrive to at the source server, repeat step 3, step 4 and step 5.

Include details on how they'll know/check?

The Destination server is now active and functional.

Step 8: Post-migration checks

As mentioned at the top, what if it's a single server going into a multi-server env? One server per domain, there were five domains. Domains that are configured to share amongst each other.

Run the following command to check for shares inconsistencies.

zxsuite backup doCheckShares

Is this an intensive/time-consuming check? If so, if the admin knows there might be share issues - would they be better off just running the doFixShares?

Should this command report any inconsistency, the

zxsuite backup doFixShares

command will parse the import mapfile used as the first argument and fix any broken share.

Mapfiles can be found in the Backup Path of the destination server as map_[source_serverID].

Could this map files also be read in human-readable format and used to "manually" recreate the shares if needed? Where the doFixShares command maybe can't do it?

Step 9: Galsync

Delete any imported GalSync accounts from the Zimbra Administration Console, then if needed create new GalSync accounts on all the imported domains and resync all the GalSync accounts with the following command:

zmgsautil forceSync -a [email protected] -n [resourcename]

Any impact to third-party clients? ZCO?

Step 10: Message Deduplication

Running a Volume Deduplication using the HSM NG module is highly suggested after a migration.

More details on this should be included or at least link to the proper section in the HSM NG guides/wiki.

What now?

  • Initialize Backup NG on the new server to make sure all of your data is safe.

Maybe reference section in 'admin guide' about Unrestorable_items ? Also about reviewing notifications/logs?

Incremental Migration FAQ

Q: Do I need a valid license in order to perform an incremental migration?

Yes. It can be either a Trial License or a purchased one.

Q: What will be migrated?

Everything except for the server configuration. This includes:

  • User Data
  • User Preferences
  • Classes of Service configuration
  • Domain configurations

Q: Will I lose my shares? Will I need to re-configure all my shares?

Absolutely not!

Q: How should I transfer the exported data between my servers?

Again, anything that suits your needs is ok. You just need to be very sure about what your *needs* are.

Do you need to move the data very fast? Physically moving an USB disk between your servers might not be a good idea.

Do you need to move the data in a very reliable way? Mounting the export folder via SSHFS to the destination server might not be a good idea if your internet connection is sloppy.