Restore and migrate

You handle three situations from the Restore page of the panel: your server runs but its data is wrong, your server is gone and a new machine replaces it, and you are replacing a working server with another. The first two use backups; the third copies a live server and has a way back until the last few minutes. Backups themselves are set up on the Backups and S3 storage page.

Prerequisites

• A whole-server restore and the receiving side of a migration work in every edition.
• Restoring a single application, the restore report, and the source side of a migration need Catena Pro or Catena Business (see the edition comparison).
• You have the panel open as an administrator.

Restore on a running server

The restore replaces data and nothing else. Applications are stopped while their data is put back, and they come back stopped: you start them yourself once you have checked them (see Start the applications after a restore). The restore of a single application leaves that application stopped too. A move to another server is the exception: its applications start with the server. The panel, the sign-in and the connection carrying the page stay up throughout, so you can follow the restore from start to finish. There is no rebuild and nothing for you to reconfigure afterwards.

1. Open Restore.

2. Under Which backups to restore from, keep This server’s backups and press Show the backups.

3. Under Choose a backup, select one row. Columns are Taken, Server, Size and Labels. Each row is a complete restore point; pick the most recent one taken before the problem started. When the repository holds backups from more than one server, the page warns that selecting by timestamp alone can restore another server’s data over this one: the Server column says which server wrote each snapshot. A backup taken since the last passing repository check, while that check fails, is labelled “Taken while the repository check fails”: it is kept, never deleted, and the label goes away for later backups once the check passes.

4. Under How much to put back, choose the scope:

   • Everything on this server returns every application to the selected backup.
   • Only the applications selected below (Catena Pro) returns only the ticked applications. Only those are stopped; every other application, the sign-in and the panel keep running. An application comes back with the data and the version recorded in the backup. Its settings are not rolled back, because they are shared with the rest of the server.

   Without a Pro subscription the scope choice is not shown, and a single-application request is refused with “Restoring a single application is not available on this plan.”

5. Under Start the restore, tick “I understand my applications will be unavailable during the restore.” The duration scales with the amount of data.

6. Press Restore the selected backup.

The option “Copy the data only, without starting anything. Used to prepare a move to another server.” copies the data and stops, leaving every application switched off. Leave it unticked for an ordinary restore. A migration does this staging by itself, so the box is not needed for one.

Progress

The Current restore section names the step in progress, in this order: Checks before anything is touched (this includes the version rule below), Recording what is running now, Stopping the applications, Copying data back, Fetching application images, Adopting the saved configuration, Starting the core services, Leaving the applications stopped, Applying the saved configuration to this server, Bringing the applications back, Restoring the databases, Bringing the applications up to date, Checking the stored files, Final checks, and Finished.

A finished restore shows a link to the section below: “The applications this restore brought back stay stopped until they are started below.”

A restore that stops part-way

The section reads “The restore stopped at:” followed by the step. Every step can be run again, so there are two ways forward:

• Carry on: choose the same backup and start it again; it continues from where it stopped. A different backup or scope is refused.
• Start over: press Clear the stopped restore, then start again from the beginning.

A second restore is refused while one runs (“A restore is already running on this server.”), and a running restore cannot be cleared.

Version rule

Each backup records the Catena version that made it. The restore compares it with the version the server runs:

• A snapshot newer than the server is refused, because newer database files do not open under an older database. Bring the server to the snapshot’s version first, then retry the restore.
• A snapshot older than the server needs an explicit upgrade choice. Without it the restore is refused and nothing is changed.
• Two versions that cannot be put in order are refused.

A refusal happens in the first step, before anything is touched.

Start the applications after a restore

A restore cannot tell whether an application’s data was compromised before the backup was taken, so the client applications it brought back stay stopped until you start them. A whole-server restore leaves every client application stopped, and a restore of a single application leaves that application stopped. Catena’s own services (the panel, the sign-in and the connection) start as before. Starting the applications works in every edition. Until you start an application, it stays stopped through updates and restarts.

While applications are left stopped, every panel page shows a banner: “Applications the restore of this server left stopped: {apps}. Start them on the Restore page once they are checked.” It links to the Applications left stopped after the restore section of the Restore page.

1. Decide whether each application can be trusted, for example that its backup predates the incident.
2. Open Restore and go to Applications left stopped after the restore. Each application has a checkbox, ticked by default, reading “Stopped since” and the date. Untick an application to leave it stopped.
3. Tick “I have checked these applications, or accept the risk of starting them”.
4. Press Start the selected applications.

Starting an application brings its databases back from the backup where they did not survive, takes it out of the mode the backup put it in, and checks its stored files. The section follows the start step by step: Waiting for the backup, the offsite copy or an update to finish, Starting the applications, Restoring the databases, Bringing the applications up to date, Checking the stored files, Final checks, and Finished. While it runs the section reads “Starting the applications:” followed by the step, and when it ends, “The last start finished.”

An application that started leaves the list. One that did not come up is stopped again and stays listed with “Its last start did not finish:” followed by the reason, such as “no container of it came up: …”, “it did not start: …” or “its stored definition names <image>, which none of its services ran when the backup was taken (…)”. The reasons are shown in English. Fix the cause, then start the application again from the same section. If the whole start stops, the section reads “Starting the applications stopped at:” followed by the step. Starting the applications again runs every step again, and an application already up is not started a second time.

Start these applications from the Restore page, not with Docker or another container manager. Starting one there starts it without its database check or its release from backup mode, and the panel warns: “Started outside this panel since the restore, without their database check or their release from backup mode: {apps}. Starting them on the Restore page runs both.” That application stays listed. Starting it from the Restore page runs what it skipped, without starting it a second time.

You can also start an application left stopped without the panel (see Leaving Catena).

Restore from another backup repository

Restore from a backup repository other than the one this server backs up to: the old server’s repository when a new machine replaces one that is gone, or a copy of this server’s backups, such as its offsite copy, when the backup repository itself is damaged, encrypted or gone. The repository is read where it is: reading it writes nothing to it, so a locked copy stays locked and a key that can only read is enough. An offsite copy keeps every backup ever copied to it, and a backup older than the copy’s lock period may be incomplete.

You need the repository’s address, its password (the backup encryption password), and an access key and secret key that can read its bucket. To rebuild a lost server, all four are in the recovery keyset (see Backups and S3 storage): first install Catena on the new server (see Installation). The installer takes the newest release.

1. Open Restore.

2. Under Which backups to restore from, choose Another backup repository. The choice is offered once the server’s installed components support it (see Troubleshooting).

3. Under Another backup repository, fill in:

   Field	Value
   Repository address	The repository address, s3:https://<endpoint>/<bucket>. The addresses of this server’s offsite copies are suggested, and so are the repositories you listed in a locked bucket.
   Repository password	The backup encryption password
   Storage access key	The S3 access key
   Storage secret key	The S3 secret key

   Only the fields the storage provider issued need values, but the address and the password are always required (“Both a repository address and its password are needed.”).

4. Press Save these credentials. They are held in memory only, and removed when a restore from that repository finishes, after 24 hours, or when the server restarts. A restore that stops part-way keeps them so you can start it again. A move to this server that has not finished 24 hours after you saved them needs them saved again. Forget the saved credentials erases them at once. The backups of that repository are then listed under Choose a backup.

5. Select the backup (after ransomware or a compromise, one taken before the incident), then continue with steps 4 to 6 of Restore on a running server. The version rule applies.

When the restore finishes, the server has the data and stored configuration of the selected backup. Its applications stay stopped until you start them (see Start the applications after a restore).

After this restore, the backups wait until you confirm the backup destination or save a new one (see Resume the backups after a restore).

Resume the backups after a restore

A whole-server restore cannot tell whether the repository it read, or the backup destination the restored configuration names, is safe to keep writing to: an attacker may hold its keys. So the backups wait after every restore of the whole server, from this server’s repository or another one. A restore of a single application and a move to another server leave the backups running.

While the backups wait:

• No backup is taken, and each backup run reports a failed attempt. The offsite copy copies nothing and the restore test is skipped.
• Every admin page shows a banner: “Backups are paused since this server was restored on {date}: a restore cannot tell whether the backup destination is safe to keep writing to. Confirm it, or save a new one, in Settings, under Backup repository.”
• The Current restore section reads “The backups wait until the backup destination is confirmed, or a new one saved, in Settings.” and offers the choice described under Choose how the backup repository carries on.

To resume them:

1. Check the repository. After a suspected compromise, prefer a new bucket with new keys.
2. Open Settings and go to Backup repository. The section reads “Backups are paused since this server was restored on {date} from {repository}. A restore cannot tell whether that repository is safe to keep writing to. Check it, then back up to it again or enter a new destination.” When the restore read a repository other than the destination the restored configuration names, the notice names both.
3. Either tick “Back up to this destination again” to keep the destination as it is, or enter the new destination and its keys.
4. Save the section.

Saving checks that the destination answers and creates a repository there when it holds none. The backups resume only when the destination opens. They keep waiting, with the reason in the section, when it does not, when no destination is saved, or while a restore is running or stopped part-way on this server (finish it or clear it on the Restore page first).

The backup password follows your choice on the Restore page:

• With A fresh repository, or with no choice and a new destination, saving makes a new backup password, shown once under Backup password further down the section. Whoever compromised the server before the restore may know the old one.
• With Continuity, the backup password stays the same.
• With no choice, confirming the same destination keeps the password.

The offsite copy that copied the old destination follows the new one. With a fresh repository it copies into a folder of its own in the same locked bucket, named after the new repository, and what it copied of the old one stays where it is. With continuity it carries on in the same folder.

With continuity, saving first opens the offsite copy and checks that it holds the same repository as the new destination. The backups keep waiting, and the section names the copy and its address, in two cases:

• The offsite copy holds another repository than the new destination. The reason ends “Put the offsite copy back into the new bucket, or choose a fresh start”. Put the offsite copy back into the new bucket first (see Put a bucket back from its offsite copy) and save that bucket again, or choose A fresh repository under How the backup repository carries on and save again.
• The offsite copy does not open, so it cannot be checked. Correct the cause the reason gives, such as the keys of the offsite copy in Settings, under Offsite copies, then save again.

Turn the nightly updates back on after a restore

For the same reason, a whole-server restore switches the nightly maintenance’s Updates and Docker engine upgrade steps off, and the other steps keep the choice you made. Nothing changes on the server overnight until you decide. Every admin page then shows a banner: “No automatic updates are configured on this server since the restore of {date}. Turn them back on in Schedules, under Nightly maintenance.” The Current restore section reads “The nightly updates stay off until they are turned back on in Schedules.”

1. Open Schedules and go to Nightly maintenance, under Steps.
2. Tick Updates, Docker engine upgrade, or both (see Choosing the steps).
3. Press Save schedules.

The banner goes once either step is on. Updates need Catena Pro or Catena Business, and the nightly maintenance itself must be turned on.

Put a bucket back from its offsite copy

This copies a bucket’s offsite copy back into the bucket it was copied from. It works for the backup repository and for any other bucket declared under Offsite copies, in every edition. The repair only adds: whatever is missing or different is copied back, and nothing is deleted on either side. The offsite copy is only read.

1. Open Restore and go to Put a bucket back from its offsite copy. The section is shown once the server’s installed components support it (see Troubleshooting). Without a declared copy it reads “No offsite copy is declared on this server. Offsite copies are declared in Settings, under Offsite copies.”
2. Under Offsite copy to put back, choose the copy. Each entry shows its name and the address of the bucket it puts back.
3. Enter Access key that reads the offsite copy and Secret key that reads the offsite copy. They are used for this run only and kept nowhere.
4. Optional: under Application that keeps its files in this bucket, choose the application when the bucket holds its files, such as Nextcloud. It is put in its backup mode while its files are copied back, then its stored files are checked against its database. Keep None for any other bucket, and for the backup repository.
5. Optional: under Into another bucket (optional), the fields start with the bucket the chosen copy is made from (Address of the new bucket) and its access key (Access key that writes the new bucket), so the files go back where they were. Leave Secret key that writes the new bucket blank to keep the secret the server holds. For the backup repository’s copy the fields start empty, because the backup repository is never put back into its own bucket. To put the copy into another, empty bucket instead, replace the address, the access key and the secret key with that bucket’s: a changed bucket or access key needs its secret. Enter Region of the new bucket (optional) when the storage provider names one: it is used for the application you chose in step 4. A bucket that holds anything is refused before the copy. With an application and a new bucket, the repair also switches the application to the new bucket (see Point Nextcloud at the new bucket).
6. Optional: under As it was at (UTC, optional), enter a date and time to put back the files as the offsite copy held them at that moment, such as before they were encrypted or overwritten. The access key then needs the right to read earlier versions. Left blank, the latest versions are put back.
7. Tick “I understand the bucket is written to while it is put back.” and press Put the bucket back.

The section follows the repair step by step: Waiting for the backup, the offsite copy or an update to finish, Reaching both buckets, Checking the bucket can take the copy back, Putting the application in its backup mode, Copying the files back, Pointing the application and the offsite copy at the new bucket, Taking the application out of its backup mode, Checking the stored files, and Finished. When you chose no application, the three steps that involve one (backup mode in, backup mode out, checking the stored files) pass without doing anything. When you named no new bucket, the pointing step passes without doing anything too.

The copy writes into the bucket while it runs. A large bucket takes hours, and the storage provider may charge for the data read.

If the repair stops, the section reads “Putting the bucket back stopped at:” followed by the step. Press Put the bucket back again with the same entries: starting again copies only what is still missing. The backup repository is never put back into its own bucket: a request to do so is refused before anything is copied. Save a new backup destination in Settings, under Backup repository, or name a new empty bucket for the copy to go into under Into another bucket (optional) and save that bucket as the backup destination (see Choose how the backup repository carries on).

Point Nextcloud at the new bucket

To move Nextcloud’s files to a new bucket, the repair form does the whole move:

1. Under Application that keeps its files in this bucket, choose Nextcloud.
2. Under Into another bucket (optional), fill Address of the new bucket, Access key that writes the new bucket and Secret key that writes the new bucket, and Region of the new bucket (optional) when the storage provider names one.
3. Fill in the rest of the form as described above and press Put the bucket back.

The repair copies the files into the new bucket. At the step Pointing the application and the offsite copy at the new bucket, it switches Nextcloud to the new bucket and its keys, renames Nextcloud’s storage so its files stay readable, and makes the offsite copy copy the new bucket from then on. Nextcloud restarts on the new bucket. If the switch fails, Nextcloud’s settings are put back. Nextcloud has to be running; a Nextcloud you stopped yourself is refused, so start it first. A Nextcloud a restore left stopped is switched when you start it from the Restore page (see Start the applications after a restore).

List the repositories in a locked bucket

To find the backup repositories an offsite copy’s locked bucket holds, such as the one a fresh start copies into its own folder:

1. Below the repair form, under Repositories in an offsite copy’s bucket, choose the copy under Offsite copy to put back.
2. Enter Access key that reads the offsite copy and Secret key that reads the offsite copy. They read the bucket for this request only and are kept nowhere.
3. Press List the repositories.

The page reads “Backup repositories in this bucket:” followed by the addresses, “This bucket holds no backup repository.” or “The bucket could not be listed.” with the reason. Each address is offered as a suggestion in Repository address under Another backup repository (see Restore from another backup repository).

Choose how the backup repository carries on

After a restore, backups wait until you confirm the backup destination, or save a new one, in Settings (see Resume the backups after a restore). While they wait, the Restore page shows How the backup repository carries on, with two choices. Press Save this choice to record yours.

• A fresh repository (selected until you choose otherwise). The backups start a new repository under a new backup password, shown once when you save or confirm the destination in Settings. Its offsite copy goes into a new folder of the same locked bucket, beside the old one. Choose this after a suspected compromise.
• Continuity. The backup repository comes back from its offsite copy into a new empty bucket that you name under Put a bucket back from its offsite copy (see the steps above, with Address of the new bucket filled). Then save that bucket as the backup destination in Settings: the backups carry on in it under the same backup password, and the offsite copy carries on in the same folder of the locked bucket, so nothing is stored twice. Choose this after an accident, such as a deleted bucket or a provider outage.

Continuity keeps the repository’s master key. A new backup password only re-wraps that key, so anyone who holds the old password and a copy of the old key file can still read the data in the continued repository if they reach its files.

Without a choice, saving a new destination makes a new backup password and confirming the same one does not. A repair and a restore do not run together: each is refused while the other runs.

Restore report

The Restore report panel (Catena Pro) shows evidence that backups restore: a local restore test with its recovery time, and the state of the offsite copy. The local test is off until you turn it on: it runs weekly on the Restore test schedule, or every night as a step of the nightly maintenance (see Schedules). It downloads as much as the backup holds and needs as much free disk, so turn it on when the server has the room. Verify my backup can be restored under Actions runs it on demand. A link to the report appears in the Current restore section.

Migrate to another server

A migration copies almost everything while the old server keeps serving. Only the last few minutes are unavailable to the people using it, and that part does not grow with the amount of data. You start the move from the new server, after the old server has opened a time-limited window.

Prerequisites

• The old server has Catena Pro or Catena Business. The new server needs no subscription of its own: the subscription key moves with the data.
• Both servers are on the same Catena version. A move between different versions is refused before anything is touched.
• The new server reaches the old one’s SSH port 22: on its public address, or on its private network once its public SSH is closed (see Admin access and tailnet).
• The old server’s backup repository is saved on the new server (steps 2 to 4 of Restore from another backup repository).
• You have not entered a Cloudflare token on the new server. The move brings the old server’s token, and a token entered first would take the old server’s web address before the move starts.

On the old server

1. Open the Migration panel.
2. Press Allow this server to be migrated. Three fields appear below the button, once: the Migration ticket, with a Copy button, the Pairing code, and Migration window closes at, the time the window closes in the server’s time zone. Neither the ticket nor the code is stored, and neither is shown again.
3. Pass both to whoever runs the move by two different channels: copy the ticket with Copy, and read the code aloud.

The window closes on its own after 4 hours, five wrong codes close it too, and Stop allowing migration closes it at once. Replace a lost ticket or code by pressing the button again, which mints a new pair and stops the old ones. The window also closes once the move has handed the subscription key over.

On the new server

1. Open Restore and go to Move another server here.
2. Paste the Migration ticket.
3. Enter the Pairing code.
4. Leave The other server’s address (optional) empty to use the address in the ticket, or enter another IP address, for example the old server’s private-network address once its public SSH is closed. Never use its web address: that is what moves at the end, so it cannot reach the old server during the move.
5. Set Preparation passes (1 to 5, default 1). Each extra pass copies only what changed since the previous one, which shortens the final unavailable period. One pass is enough unless you expect a long delay before the cut-over.
6. Tick “I understand the other server will be taken out of service and that this one will take over its web address.” and press Start the move.

What happens

1. Reaching the other server.
2. Copying the data across, and Getting this server ready to receive traffic. The old server still serves.
3. Pausing the other server’s applications. The short unavailable window begins.
4. Taking its final backup, then Checking that backup. This is the last reversible step.
5. Taking the other server out of service. From here the old server does not put itself back.
6. Putting the data in place here, then Moving the web address here (the cut-over).
7. Final checks.

A move starts the applications with the server: nothing is left stopped.

After the final checks the old server frees its activation of the subscription key, the new server activates it and then brings itself up to date, which turns on what the key unlocks (extra domains, schedules). Some services restart during that. Settings > Subscription shows the result. If freeing the old activation fails, free it in Polar’s customer portal and save the key again on the new server.

Calling it off

• Up to and including “Checking that backup”, stopping the move puts the old server back into service by itself. Nothing has started on the new server and the web address still points at the old one.
• After “Taking the other server out of service”, that no longer happens on its own, because the new server may already hold part of the data and starting both would have two servers writing to the same storage. The page reports that the old server is out of service and offers two ways: fix what failed and start the move again (it continues without copying everything a second time), or put the old server back.
• To put the old server back, use Put it back into service under Put the other server back into service on the new server’s Restore page (enter the ticket, the pairing code and, if needed, the address again), or Put this server back into service in the old server’s own Migration panel. Its web address still points there, so it serves again as soon as it comes up.
• Forget this move clears the record of an unfinished move and changes nothing on either server.

After a completed move the old server is stopped, not erased. Its data and backups are untouched, so you can keep it until the new server has proven itself. Retiring it is a separate step that nothing in the move performs.

Troubleshooting

Message	Meaning
“Choose a backup to restore.”	You have not selected a row.
“Confirm that applications may be unavailable before starting.”	You have not ticked the confirmation box.
“Select at least one application, or choose to restore everything.”	The scope is limited to applications and you have ticked none.
“A restore is running on this server. Wait for it to finish before moving another server here.”	A migration cannot start during a restore.
“A move is already running on this server.”	Wait for it, or use Forget this move if it is unfinished.
“The move could not be started.”	The request was not accepted. Check the ticket, the pairing code and the address (an IP address, not a name), and that the window is open on the old server.
The source refused the ticket: its window is closed, it was armed again since, or the ticket is not this server’s	The old server did not accept the ticket. Open a new window on the old server, then use its new ticket and code.
The server at that address is not the one that minted the ticket	The address reaches another server, or the old server’s SSH host key changed after a reinstall or a restore. Check the address; after a reinstall or a restore, open a new window on the old server and use its new ticket.
“The credentials were refused.”	The repository details were not accepted. Check them against the recovery keyset.
The list of backups reads “this server’s backup repository refused its access key: the key is wrong or revoked.”	The storage provider refused the access key this server holds. The page answers within about 20 seconds instead of waiting. Correct the key in Settings, under Backup, or enter the keyset of the repository to restore from under Another backup repository. The same list can also read that the repository did not answer, does not exist, or does not open with the backup password. The reason is shown in English.
“A bucket is being put back from its offsite copy. Wait for it to finish before starting a restore.”	A bucket repair is running. A restore starts only after it finishes, and a repair is refused the same way while a restore runs (“A restore is running on this server. Wait for it to finish before putting a bucket back.”).
“Restoring from another backup repository is not available on this server yet: its installed components are older than this panel. Bring this server up to date in Settings, under Server configuration, then try again.”, “Putting a bucket back is not available on this server yet: its installed components are older than this panel.” or “Starting the applications from this page is not available on this server yet: its installed components are older than this panel. Bring this server up to date in Settings, under Server configuration, then try again.”	The server’s installed components predate the feature. Open Settings, go to Server configuration, press Bring this server up to date, and try again once it finishes.
“Confirm that these applications were checked, or that the risk of starting them is accepted, before starting them.”	You have not ticked the confirmation box in Applications left stopped after the restore.
“A restore is running on this server. Wait for it to finish before starting the applications.”	Applications cannot be started during a restore.
“A bucket is being put back from its offsite copy. Wait for it to finish before starting the applications.”	A bucket repair is running. Start the applications after it finishes.
“The applications are already being started on this server.”	A start is in progress. Wait for it to finish.
“The applications could not be started.”	The request was refused. The server’s reason follows, such as “choose at least one application to start” or an application that is no longer left stopped.
“Its last start did not finish:”	The reason follows. The application was stopped again and stays listed. Fix the cause, then start it again from Applications left stopped after the restore.
“Started outside this panel since the restore, without their database check or their release from backup mode: …”	The listed applications were started with Docker or another container manager. Start them from the Restore page to run the database check and the release from backup mode they skipped.
“The backups still wait: …” or “The server did not answer, so the backups still wait. Saving the destination again retries.” under Backup repository	The destination was saved but the backups did not resume. The reason follows: the destination does not open, no destination is saved, a restore is running or stopped part-way, or, with continuity, the offsite copy holds another repository or does not open (see Resume the backups after a restore). Fix the cause, then save again.
“No new backup password was made, so the backups still wait: …”	The new backup password could not be made. Save the destination again.
Log: the restore left <app> stopped, or the Restore page reads “its stored definition names <image>, which none of its services ran when the backup was taken (<images>)”	The application’s restored compose file names an image that none of its services ran when the backup was taken. On the server, as root over SSH, edit /srv/catena/compose/<app>/docker-compose.yml so that each image it names is one listed in the parentheses; an image set through a variable counts as the value in the folder’s .env. Then start the application from the Restore page.