Data Migrations
- on a larger note we have two possible scenarios
- one-time migration
- on-going sync
One-Time Migration
- Determine size and calculate the time to transfer
- If the time taken to upload is more consider offline options
- Azure options for offline storage transfer: Azure Databoxc
- AWS options for offline storage transfer: AWS Snowball
AWS Snow devices: overview
AWS provides physical appliances for moving large volumes of data when network transfer is too slow, unreliable or too expensive. Key devices are Snowcone, Snowball Edge, Snowmobile. ([Amazon Web Services, Inc.][1])
Below I describe each, with specs and use-cases.
Device types & specs
| Device | Usable Capacity / Typical Sizes | Compute / Memory (if any) | Physical / Portability & Ruggedness | Best use cases |
|---|---|---|---|---|
| AWS Snowcone | Up to ~8 TB of storage. ([KodeKloud Notes][2]) | Minimal compute; meant for very portable or constrained-power/space scenarios. ([AWS Static][3]) | Very small, rugged, can work in remote locations, limited power, etc. ([AWS Static][3]) | When data volumes are small (tens of TBs or less), or in remote/edge locations; maybe collecting data at remote sites or IoT, then shipping the device physically. |
| AWS Snowball Edge (Storage Optimized, Compute Optimized) | Storage-Optimized: ~210 TB usable NVMe storage. ([AWS Documentation][4]) Compute-Optimized: much less storage (e.g. ~28 TB NVMe) but with high CPU/RAM. ([AWS Documentation][4]) |
Compute-Optimized: up to ~104 vCPUs, ~416 GB RAM. ([AWS Documentation][4]) Storage-Optimized also has full CPU/memory but workloads are more focused on storage rather than compute. ([AWS Documentation][4]) |
Heavier, more rugged; still portable but needs good logistics. Ships in a rugged chassis. ([AWS Documentation][4]) | Bulk data center migrations, large archive moves, when you also want to pre-process data locally (e.g. filter, transform, compress) before shipping it to the cloud. |
| AWS Snowmobile | Up to 100 petabytes (PB) per unit. ([CloudOptimo][5]) | Huge compute facility capacity but mostly used just for shipping data; you’d orchestrate in/out pre/post on your side. | It’s essentially a shipping container / mobile data center on a truck. Very large, requires special logistics. ([CloudOptimo][5]) | When your data is extremely large (many tens of PBs or 100PB scale), and network or even multiple Snowball devices would be too slow or too cumbersome. Data center evacuations or migration of huge archives. |
Security, transfer & operational details
- Data on all these devices is encrypted (AES-256) and keys managed via AWS KMS. ([AWS Static][3])
- Devices are rugged and tamper-evident. “Tamper-evident box”, intrusion detection, etc. ([Amazon Web Services, Inc.][1])
- Connectivity: when the device is on site, you copy data onto it either via network protocols (NFS, SMB) or via the S3-compatible interface that the device provides. ([Amazon Web Services, Inc.][6])
- After copying, you ship the device back to AWS. AWS then moves the data into your S3 buckets (or other designated target), verifies it, then securely erases the data from the device. ([Amazon Web Services, Inc.][1])
How to choose the right device
Some key criteria to decide:
-
Data volume: how much TB/PB of data need to move?
- < ~10 TB → maybe Snowcone is sufficient
- Tens to few hundreds TB → Snowball Edge Storage-Optimized
- Multi-PB → Snowmobile
-
Compute / preprocessing needs: If you need to do filtering, compression, transformation, analytics before shipping, then a compute-optimized device helps.
-
Physical constraints & logistics: weight, shipping, power, environmental conditions. If remote site, harsh environment, limited power, then lean toward smaller/ruggeder devices.
-
Timeline vs Network Capacity: Calculate how long would it take to transfer over your network vs shipping. Sometimes offline (physical) shipping is far faster than online transfer when bandwidth is low or unreliable.
-
Cost: Rental fees, shipping, retention period of device, etc. Also operational cost of moving data locally onto the device (e.g. staging machines, manpower).
Ongoing transfers
- Azure has Azure Databox Gateway which can be setup on Hyperv or Vmware and configured to copy every day transfers to cloud
- AWS Storage Gateway
