How To: Backblaze B2 as a Proxmox Backup Server 4 S3 Datastore

Before deciding on going all in on cloud storage for your Proxmox backups, you need to understand potential cost implications. Different cloud providers bill differently for ingres, egress, and API calls. Understanding how Proxmox Backup Server uses S3 storage is key to understanding the cost implications for your cloud provider.

I got a bit of a nasty surprise bill from BackBlaze B2, which increased my typical monthly bill of $5 to $25. When reviewing my billing details, it was apparent that the surprise costs were from egress fees and excessive API calls. 

I inquired about this on the Proxmox forums, as it appear like the verification job I had running was potentially the problem child. Chris (a Proxmox Staff Member) responded and had this to say:

This is expected, I’ll try to explain what is going on here. The backup protocol is optimized to only upload the necessary data from the client to the server, and from the server to the S3 backend. The client (PVE) will use the fast incremental mode if possible to avoid re-reading and re-uploading unchanged data blocks (chunks). Only these which changed are uploaded and re-indexed, others re-indexed only. For security reasons, the client is only allowed to re-index and skip upload for chunks it has uploaded in the previous backup snapshot, it will re-upload others. The server then checks if that chunk is already known and present on its local datastore cache, avoiding re-upload when possible. This helps to reduce upload requests and bandwidth to the S3 API. Data consistency when uploading to the S3 backend is further assured by check-summing, so it can be assured the data the server has send is also persisted to the S3 object store.
If you now perform a verification of the backup snapshot, the index file referencing all the data chunks will be used to identify all objects which belong to this snapshot and fetch them from the object store, not just the newly uploaded. Once fetched, the chunk data is verified to assure that the data it stores is the data it must contain. This is analogous to what happens on regular datastores. So if you are verifying after each backup snapshot, you are effectively downloading all data chunks just as if you would do a full restore. There is room for improvements regarding the verification process performance and cost effectiveness with ideas floating around on how to do that, but not implemented yet.

For the time being a verification will always fetch the chunk objects from S3, unconditionally. So I would recommend to strongly reduce the frequency at which you do verifications, and to do verifications in bigger patches, not just single snapshots. This is since the verification job has an optimization which avoids re-download and re-verification of already verifyied chunks for the same verification job.

So there you have it, S3 verification jobs can incur a large amount of downloaded data and API costs. Just like my local NAS datastores, I had a nightly S3 verification job that ran after all of my backups were complete. Based on Chris’s information, I would look at reducing the frequency to maybe once or twice a month, depending on your needs and quantity of data.