This page describes how to use the managed import and export features to move Cloud Firestore data from one project to another. This can be useful for setting up a development environment or as part of permanently migrating an app to another project. The example on this page demonstrates how to export data from a source project and then import that data into a destination project. Moving data between projects involves the following steps:
- Create a Cloud Storage bucket to hold the data from your source project.
- Export the data from your source project to the bucket.
- Give your destination project permission to read from the bucket.
- Import the data from the bucket into your destination project.
Before you begin
Before you can use the managed export and import service, you must complete the following tasks:
- Enable billing for both your source project and destination project. Only Google Cloud projects with billing enabled can use the export and import functionality.
-
Make sure your account has the necessary Cloud IAM permissions in your source project and destination project. If you are a project owner for both projects, your account has the required permissions. Otherwise, the following Cloud IAM roles grant the necessary permissions for Cloud Firestore export and import operations:
Owner
,Cloud Datastore Owner
, orCloud Datastore Import Export Admin
A project owner can grant one of these roles by following the steps in Grant access.
-
Set up the
gcloud
command-line tool and connect to your project in one of the following ways:-
Access
gcloud
from the Google Cloud console using Cloud Shell.Make sure
gcloud
is configured for the correct project:gcloud config set project [SOURCE_PROJECT_ID]
-
-
Set up indexes in your new project. The composite indexes should match between the source and destination projects. Indexes should be set up first to avoid having to process each document multiple times.
Export data from the source project
Export your data by creating a Cloud Storage bucket for your Cloud Firestore export files and starting an export operation.
Create a Cloud Storage bucket
Create a Cloud Storage bucket in the same location as your Cloud Firestore database. To view your database location, see your project location setting. You cannot use a Requester Pays bucket for export and import operations.
If your Cloud Storage bucket is not in your
source project, you must give the source project's
default service account access to the bucket. Each Google Cloud project
has an automatically created default service account with the name PROJECT_ID@appspot.gserviceaccount.com
. Cloud Firestore
export operations use this default service account to authorize Cloud Storage
bucket operations. To give the
default service account access to your source bucket, grant it the
Storage Admin
role.
You can grant this role with the
gsutil
tool
available in Cloud Shell:
gsutil iam ch serviceAccount:[service-PROJECT_NUMBER]@gcp-sa-firestore.iam.gserviceaccount.com :roles/storage.admin\ gs://[BUCKET_NAME]@
You can also grant this role in the Google Cloud console.
Disable write operations (optional)
If your app continues to write to your database while you perform an export operation, you might not capture all of those writes in your export files. To export data from a consistent state, disable writes to your database by updating your security rules and halting any Admin SDK operations.
Update security rules
In the Cloud Firestore Rules tab of the console, update your source project security rules to deny all writes. For example:
// Deny write access to all users under any conditions service cloud.firestore { match /databases/{database}/documents { match /{document=**} { allow write: if false; } // Reads do not affect export operations // Add your read rules here } }
Halt writes from Admin SDKs
Security rules do not stop writes coming from privileged server environments created using a Firebase Admin SDK or a Google Cloud Server Client Library. Make sure to halt write operations from your admin servers by shutting down or updating your servers.
Begin an export operation
Use the gcloud firestore export
command to export
data from your source project. You can export all your data or only
specific collections. Replace [SOURCE_BUCKET]
with the name of your
Cloud Storage bucket:
- Export all data
-
gcloud firestore export gs://[SOURCE_BUCKET] --async
- Export specific collections
-
gcloud firestore export gs://[SOURCE_BUCKET] --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2] --async
Take note of your export operation's outputURIPrefix
as you will use
this later on. By default, Cloud Firestore adds a pre-fix to your export
files based on a timestamp:
outputUriPrefix: gs://[SOURCE_BUCKET]/2019-03-05T20:58:23_56418
As the export operation runs, you can use the firestore operations list
command to view your operation's progress:
gcloud firestore operations list
Import data into the destination project
Next, give the destination project access to your Cloud Firestore data files and start an import operation.
Give the destination project access to your data files
Before you can begin an import operation, you must make sure your destination project can access your Cloud Firestore data files.
Move data files to a local bucket
If your source bucket location is different from the Cloud Firestore location of your destination project, you must move your data files to a Cloud Storage bucket in the same location as your destination project.
Move your data files to another Cloud Storage bucket by following the steps
in Moving and Renaming Buckets.
For all the following steps, use this new bucket as the [SOURCE_BUCKET]
.
Give the project service account access to your source bucket
If your source bucket is not in your destination project, then you must give the
destination project's default service account access to your source bucket. The
default service account is named
[DESTINATION_PROJECT_ID]@appspot.gserviceaccount.com
. To
give the default service account access to your source bucket, grant it the
proper permissions to access the bucket.
You can grant the necessary roles with the
gsutil
tool
available in Cloud Shell:
gsutil iam ch serviceAccount:[DESTINATION_PROJECT_ID]@appspot.gserviceaccount.com:legacyBucketReader,legacyObjectReader \ gs://[SOURCE_BUCKET]
You can also grant this role in the Google Cloud console.
Begin an import operation
Before starting the import operation, make sure gcloud
is configured
for the correct project:
gcloud config set project [DESTINATION_PROJECT_ID]
Use the gcloud firestore import
command to import the data in
your source bucket into your destination project:
gcloud firestore import gs://[SOURCE_BUCKET]/[EXPORT_PREFIX] --async
Where [EXPORT_PREFIX]
matches the pre-fix in your export operation's
outputUriPrefix
. For example:
gcloud firestore import gs://[SOURCE_BUCKET]/2019-03-05T20:58:23_56418 --async
As the export operation runs, you can use the firestore operations list
command to view your operation's progress:
gcloud firestore operations list