New to Proxmox Backup Server (PBS) 4.0 is a ‘tech preview’ feature for using S3 buckets as a datastore. While many people associate S3 buckets with Amazon, it is more or less a standard that many cloud providers support. This opens up many possibilities for direct PBS to cloud backups, which is great news.
However, I cannot emphasize enough this new feature IS ‘tech preview’ and I’ve seen issues with it. Namely, backup job verifications often fail on bad chunks of data. So I would in NO WAY (as of PBS 4.0.14) rely on S3 backups as being reliable and bulletproof. However, I imagine the feature will mature and the bugs will be worked out over time.
As such, this blog post is more a FYI on how to configure S3 storage and is fun to do out of curiosity. But until PBS works out the bugs, I would not rely on it. My preferred S3 compatible cloud storage is Backblaze B2. Unlike Wasabi, there’s no minimum billing retention period. And unlike other providers, there’s no egress fees (subject to plan limits).Â
Backblaze B2 is priced at $6/TB/month which is VERY reasonable. You are billed only for the amount of storage you use. Buckets can also be deleted and the billing for that storage amount immediately stops (unlike Wasabi).Â
If you are not using Backblaze B2, the Proxmox team has documented a few other S3 configurations, such as Cloudflare R2. Check out their post here.
Related Synology Posts
As part of my Proxmox Backup Server 4.0 and Proxmox VE 9.0 series, check out these related posts:
Create Backblaze S3 Bucket
- Login to your Backblaze account and under B2 Cloud Storage, click on Buckets.
- At the top of the screen click on Create a Bucket.
- Give the bucket a globally unique name. Leave the rest of the default values, unless you wish otherwise.
- Click Create a Bucket.
Note: Proxmox Backup Server appears to be VERY picky about bucket names that it considers acceptable. Backblaze lets me use upper case, lower case, numbers, and some special characters. However, PBS was rejecting some of them:
parameter verification errors (400)
backend: schema validation failed: Bucket name does not match the regex pattern
I suggest using all lowercase letters and digits. No upper case and no special characters.Â
5. In the left pane under B2 Cloud Storage, click Application Keys.
6. Click on Add a New Application Key.
7. Choose a name for the key, select your PBS bucket, then tick the box to allow listing of all buckets.
8. Click Create New Key.
8. Safely store the keyID and applicationKey. The applicationKey will NEVER be shown again.Â
9. Go back to your newly created bucket and click on Lifecycle Settings.
10. I suggest changing the setting to Keep only the last version of the file.Â
11. Review the Endpoint properties of your bucket and locate the region (between ‘s3.’ and ‘.backblazeb2.com’.) and copy it to the clipboard. Mine is us-west-004.Â
Add Proxmox Backup Server S3 Endpoint
Note: The PBS S3 Endpoint is designed to support multiple buckets. So you MUST use the string ‘{{bucket}}.s3.{{region}}.backblazeb2.com’ as the Endpoint, and do NOT input the FQDN of your bucket.Â
- Login to your Proxmox Backup Server (PBS) 4.0 (or later).
- On the left click on S3 Endpoints.
- Click on Add. Use the following parameters:
- S3 Endpoint ID: Any name you wish
- Endpoint: {{bucket}}.s3.{{region}}.backblazeb2.com [This exact string.]
- Port: Leave default
- Region: Your Backblaze bucket region (e.g. us-west-004)
- Access Key: Paste your keyID
- Secret Key: Paste your applicationKey
- Path Style:Â Leave UN-checked
- Provider Quirks: Select Skip If-None-Match header
4. Click Add.
Note: If you get an error such as the one below, double check your endpoint doesn’t have any leading or trailing spaces.Â
parameter verification errors (400)
endpoint: value does not match the regex pattern
Add Proxmox Backup Server S3 Datastore
Note: PBS uses local disk space to cache some of the S3 chunks. Proxmox strongly suggests adding a dedicated partition to your PBS server that is 64-128GB for the cache. If your PBS server is a VM, simply add another virtual disk, partition, format, and mount it.Â
- In the left pane under Datastore click Add Datastore.
- Use the following parameters:
- Name: Any datastore name you wish
- Datastore Type: S3 (tech preview)
- Local Cache: /mnt/backblazecache (your new partition)
- S3 Endpoint ID: Your configured endpoint name
- GC Schedule: daily
- Prune Schedule: daily
- Bucket: Your Backblaze B2 bucket
3. Click on Prune Options.
4. Configure reasonable retention periods for your Backblaze B2 bucket. Keep costs in mind when considering how long to keep your data.
5. Click Add. Wait for the datastore to be successfully created.
6. If you have created a separate ‘backup’ user (used by Proxmox VE hosts to connect to your PBS server), we need to adjust the permissions on the datastore.Â
7. Locate your Backblaze datastore and click on it. In the right pane click on Permissions. Click Add -> User Permission.
8. Select the backup user account (e.g. backup@pbs), then give it the DatastoreAdmin role. Click Add.
Add PBS Datastore to Proxmox VE
Now that PBS has been configured with your Backblaze B2 datastore, we need to mount it on your Proxmox VE hosts.
- Login to the Proxmox VE host that will be running your backup jobs.
- Change to Server View (top left), then click on the Datacenter object.Â
- In the middle pane click on Storage.
- In the right pane click on Add, and from the drop down select Proxmox Backup Server.
- Enter the following information:
- ID: Any name you wish (e.g. PBS-Backblaze)
- Server: IP or FQDN of your PBS server
- Username: Backup account username (don’t use root)
- Password: Backup account password
- Datastore: Name of the datastore on the PBS server
- Fingerprint: Only needed if using untrusted certificate
Note: If your PBS server has untrusted SSL certificates, you will need the fingerprint of your PBS server. Go back to your PBS server, click on Dashboard in the upper left, then click on Show Fingerprint on the right. Copy it to the clipboard.
6. If you want your backups encrypted in the cloud (STRONGLY recommended), click on the Encryption tab.Â
7. Select Auto-generate a client encryption key, download it, and store it in a safe place.Â
8. Click Add to add the datastore.
Add Verify Job (Optional)
I strongly suggest creating a verify job to check the integrity of your S3 bucket. During the tech preview I’ve seen verify jobs failing due to bad chunks.
- Click on the Backblaze datastore in the left pane.
- Click on the Verify Jobs tab.
- Click Add.
- Use all the defaults but I suggest changing the schedule to run a few hours after your backups are expected to complete. This will use internet bandwidth, so be mindful. I chose 06:00.
- Click Add.
Final Configuration
Now that your PBS datastore is mounted to your Proxmox VE host, you can configure your VM and LXC backup jobs. For more details on these steps, check out the second half of my article: How To: Proxmox Backup Server 4 (VM) Installation.
Performance Observations
After I configured my B2 datastore, I did a fresh backup of my primary Proxmox VE server. My internet speed is 1 Gbps symmetrical, and the backup job averaged 94MB/s (yes, bytes). The 110GB backup job completed in 20 minutes, which is astounding.
However, I immediately ran a verification job of that same 110GB bucket, and it took 2 hours and 24 minutes. 7x slower to verify than to backup. I’m not sure if this performance difference is expected or not, but it’s what I observed.
Summary
Brand new to Proxmox Backup Server 4.0 is the ability to use S3 compatible storage buckets. In this blog post I walk you through how to configure a Backblaze B2 bucket for use with Proxmox Backup Server 4.0. We then mount it as a backup datastore to Proxmox VE, and schedule backup jobs against it.Â
This feature is in ‘tech preview’ and I’ve seen backup verification jobs fail due to bad chunks. So I would NOT rely on this as a backup destination until it comes out of ‘tech preview’ and the bugs are resolved. Backup speeds were able to saturate my 1 Gbps symmetrical internet. However, a verification job took 7x longer to complete.
