JamBlog

• Method Details

  • downloadFile

    Downloads an object identified by the bucket and key from S3 to a local file. For non-file-based downloads, you may use download(DownloadRequest) instead.

    The SDK will create a new file if the provided one doesn't exist. The default permission for the new file depends on the file system and platform. Users can configure the permission on the file using Java API by themselves. If the file already exists, the SDK will replace it. In the event of an error, the SDK will NOT attempt to delete the file, leaving it as-is.

    Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

    Usage Example:

    Copy

     S3TransferManager transferManager = S3TransferManager.create(); DownloadFileRequest downloadFileRequest = DownloadFileRequest.builder() .getObjectRequest(req -> req.bucket("bucket").key("key")) .destination(Paths.get("myFile.txt")) .addTransferListener(LoggingTransferListener.create()) .build(); FileDownload download = transferManager.downloadFile(downloadFileRequest); // Wait for the transfer to complete download.completionFuture().join(); 

    See Also:

  • downloadFile

    See Also:

  • resumeDownloadFile

    Resumes a downloadFile operation. This download operation uses the same configuration as the original download. Any content that has already been fetched since the last pause will be skipped and only the remaining data will be downloaded from Amazon S3.

    If it is determined that the source S3 object or the destination file has be modified since the last pause, the SDK will download the object from the beginning as if it is a new DownloadFileRequest.

    Usage Example:

    Copy

     S3TransferManager transferManager = S3TransferManager.create(); DownloadFileRequest downloadFileRequest = DownloadFileRequest.builder() .getObjectRequest(req -> req.bucket("bucket").key ("key")) .destination(Paths.get("myFile.txt")) .build(); // Initiate the transfer FileDownload download = transferManager.downloadFile(downloadFileRequest); // Pause the download ResumableFileDownload resumableFileDownload = download.pause(); // Optionally, persist the download object Path path = Paths.get("resumableFileDownload.json"); resumableFileDownload.serializeToFile(path); // Retrieve the resumableFileDownload from the file resumableFileDownload = ResumableFileDownload.fromFile(path); // Resume the download FileDownload resumedDownload = transferManager.resumeDownloadFile(resumableFileDownload); // Wait for the transfer to complete resumedDownload.completionFuture().join(); 

    Parameters: resumableFileDownload - the download to resume. Returns: A new FileDownload object to use to check the state of the download. See Also:

  • resumeDownloadFile

    See Also:

  • download

    Downloads an object identified by the bucket and key from S3 through the given AsyncResponseTransformer. For downloading to a file, you may use downloadFile(DownloadFileRequest) instead.

    Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

    Usage Example (this example buffers the entire object in memory and is not suitable for large objects):

    Copy

     S3TransferManager transferManager = S3TransferManager.create(); DownloadRequest<ResponseBytes<GetObjectResponse>> downloadRequest = DownloadRequest.builder() .getObjectRequest(req -> req.bucket("bucket").key("key")) .responseTransformer(AsyncResponseTransformer.toBytes()) .build(); // Initiate the transfer Download<ResponseBytes<GetObjectResponse>> download = transferManager.download(downloadRequest); // Wait for the transfer to complete download.completionFuture().join(); 

    See the static factory methods available in AsyncResponseTransformer for other use cases.

    Type Parameters: ResultT - The type of data the AsyncResponseTransformer produces Parameters: downloadRequest - the download request, containing a GetObjectRequest and AsyncResponseTransformer Returns: A Download that can be used to track the ongoing transfer See Also:

  • uploadFile

    Uploads a local file to an object in S3. For non-file-based uploads, you may use upload(UploadRequest) instead.

    Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners. Upload a local file to an object in S3. For non-file-based uploads, you may use upload(UploadRequest) instead.

    Usage Example:

    Copy

     S3TransferManager transferManager = S3TransferManager.create(); UploadFileRequest uploadFileRequest = UploadFileRequest.builder() .putObjectRequest(req -> req.bucket("bucket").key("key")) .addTransferListener(LoggingTransferListener.create()) .source(Paths.get("myFile.txt")) .build(); FileUpload upload = transferManager.uploadFile(uploadFileRequest); upload.completionFuture().join(); 

    See Also:

  • uploadFile

    See Also:

  • resumeUploadFile

    Resumes uploadFile operation. This upload operation will use the same configuration provided in ResumableFileUpload. The SDK will skip the data that has already been upload since the last pause and only upload the remaining data from the source file.

    If it is determined that the source file has be modified since the last pause, the SDK will upload the object from the beginning as if it is a new UploadFileRequest.

    Usage Example:

    Copy

     S3TransferManager transferManager = S3TransferManager.create(); UploadFileRequest uploadFileRequest = UploadFileRequest.builder() .putObjectRequest(req -> req.bucket("bucket").key("key")) .source(Paths.get("myFile.txt")) .build(); // Initiate the transfer FileUpload upload = transferManager.uploadFile(uploadFileRequest); // Pause the upload ResumableFileUpload resumableFileUpload = upload.pause(); // Optionally, persist the resumableFileUpload Path path = Paths.get("resumableFileUpload.json"); resumableFileUpload.serializeToFile(path); // Retrieve the resumableFileUpload from the file ResumableFileUpload persistedResumableFileUpload = ResumableFileUpload.fromFile(path); // Resume the upload FileUpload resumedUpload = transferManager.resumeUploadFile(persistedResumableFileUpload); // Wait for the transfer to complete resumedUpload.completionFuture().join(); 

    Parameters: resumableFileUpload - the upload to resume. Returns: A new FileUpload object to use to check the state of the download. See Also:

  • resumeUploadFile

    See Also:

  • upload

    Uploads the given AsyncRequestBody to an object in S3. For file-based uploads, you may use uploadFile(UploadFileRequest) instead.

    Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

    Usage Example:

    Copy

     S3TransferManager transferManager = S3TransferManager.create(); UploadRequest uploadRequest = UploadRequest.builder() .requestBody(AsyncRequestBody.fromString("Hello world")) .putObjectRequest(req -> req.bucket("bucket").key("key")) .build(); Upload upload = transferManager.upload(uploadRequest); // Wait for the transfer to complete upload.completionFuture().join(); 

    See the static factory methods available in AsyncRequestBody for other use cases. Parameters: uploadRequest - the upload request, containing a PutObjectRequest and AsyncRequestBody Returns: An Upload that can be used to track the ongoing transfer See Also:

  • upload

    See Also:

  • uploadDirectory

    Uploads all files under the given directory to the provided S3 bucket. The key name transformation depends on the optional prefix and delimiter provided in the UploadDirectoryRequest. By default, all subdirectories will be uploaded recursively, and symbolic links are not followed automatically. This behavior can be configured in at request level via UploadDirectoryRequest.Builder.followSymbolicLinks(Boolean) or client level via S3TransferManager.Builder.uploadDirectoryFollowSymbolicLinks(Boolean) Note that request-level configuration takes precedence over client-level configuration.

    By default, the prefix is an empty string and the delimiter is "/". Assume you have a local directory "/test" with the following structure:

      |- test |- sample.jpg |- photos |- 2022 |- January |- sample.jpg |- February |- sample1.jpg |- sample2.jpg |- sample3.jpg  

    Give a request to upload directory "/test" to an S3 bucket, the target bucket will have the following S3 objects:
    • sample.jpg
    • photos/2022/January/sample.jpg
    • photos/2022/February/sample1.jpg
    • photos/2022/February/sample2.jpg
    • photos/2022/February/sample3.jpg

    The returned CompletableFuture only completes exceptionally if the request cannot be attempted as a whole (the source directory provided does not exist for example). The future completes successfully for partial successful requests, i.e., there might be failed uploads in the successfully completed response. As a result, you should check for errors in the response via CompletedDirectoryUpload.failedTransfers() even when the future completes successfully.

    The current user must have read access to all directories and files.

    Usage Example:

    Copy

     S3TransferManager transferManager = S3TransferManager.create(); DirectoryUpload directoryUpload = transferManager.uploadDirectory(UploadDirectoryRequest.builder() .source(Paths.get("source/directory")) .bucket("bucket") .s3Prefix("prefix") .build()); // Wait for the transfer to complete CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join(); // Print out any failed uploads completedDirectoryUpload.failedTransfers().forEach(System.out::println); 

    Parameters: uploadDirectoryRequest - the upload directory request See Also:

  • uploadDirectory

    See Also:

  • downloadDirectory

    Downloads all objects under a bucket to the provided directory. By default, all objects in the entire bucket will be downloaded. You can modify this behavior by providing a DownloadDirectoryRequest.listObjectsRequestTransformer() and/or a DownloadDirectoryRequest.filter() in DownloadDirectoryRequest to limit the S3 objects to download.

    The downloaded directory structure will match with the provided S3 virtual bucket. For example, assume that you have the following keys in your bucket:

    • sample.jpg
    • photos/2022/January/sample.jpg
    • photos/2022/February/sample1.jpg
    • photos/2022/February/sample2.jpg
    • photos/2022/February/sample3.jpg
    Give a request to download the bucket to a destination with path of "/test", the downloaded directory would look like this

      |- test |- sample.jpg |- photos |- 2022 |- January |- sample.jpg |- February |- sample1.jpg |- sample2.jpg |- sample3.jpg  

    The returned CompletableFuture only completes exceptionally if the request cannot be attempted as a whole (the downloadDirectoryRequest is invalid for example). The future completes successfully for partial successful requests, i.e., there might be failed downloads in a successfully completed response. As a result, you should check for errors in the response via CompletedDirectoryDownload.failedTransfers() even when the future completes successfully.

    The SDK will create the destination directory if it does not already exist. If a specific file already exists, the existing content will be replaced with the corresponding S3 object content.

    The current user must have write access to all directories and files

    Usage Example:

    Copy

     S3TransferManager transferManager = S3TransferManager.create(); DirectoryDownload directoryDownload = transferManager.downloadDirectory( DownloadDirectoryRequest.builder() .destination(Paths.get("destination/directory")) .bucket("bucket") // only download objects with prefix "photos" .listObjectsV2RequestTransformer(l -> l.prefix("photos")) .build()); // Wait for the transfer to complete CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join(); // Print out any failed downloads completedDirectoryDownload.failedTransfers().forEach(System.out::println); 

    Parameters: downloadDirectoryRequest - the download directory request See Also:

  • downloadDirectory

    See Also:

  • copy

    Creates a copy of an object that is already stored in S3.

    Depending on the underlying S3Client, S3TransferManager may intelligently use plain CopyObjectRequests for smaller objects, and multiple parallel UploadPartCopyRequests for larger objects. If multipart copy is supported by the underlying S3Client, this behavior can be configured via S3CrtAsyncClientBuilder.minimumPartSizeInBytes(Long). Note that for multipart copy request, existing metadata stored in the source object is NOT copied to the destination object; if required, you can retrieve the metadata from the source object and set it explicitly in the @link CopyObjectRequest.Builder#metadata(Map)}.

    While this API supports TransferListeners, they will not receive bytesTransferred callback-updates due to the way the CopyObjectRequest API behaves. When copying an object, S3 performs the byte copying on your behalf while keeping the connection alive. The progress of the copy is not known until it fully completes and S3 sends a response describing the outcome.

    If you are copying an object to a bucket in a different region, you need to enable cross region access on the S3AsyncClient.

    Usage Example:

    Copy

     S3AsyncClient s3AsyncClient = S3AsyncClient.crtBuilder() // enable cross-region access, only required if you are making cross-region copy .crossRegionAccessEnabled(true) .build(); S3TransferManager transferManager = S3TransferManager.builder() .s3Client(s3AsyncClient) .build(); CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder() .sourceBucket("source_bucket") .sourceKey("source_key") .destinationBucket("dest_bucket") .destinationKey("dest_key") .build(); CopyRequest copyRequest = CopyRequest.builder() .copyObjectRequest(copyObjectRequest) .build(); Copy copy = transferManager.copy(copyRequest); // Wait for the transfer to complete CompletedCopy completedCopy = copy.completionFuture().join(); 

    Parameters: copyRequest - the copy request, containing a CopyObjectRequest Returns: A Copy that can be used to track the ongoing transfer See Also:

  • copy

    See Also:

  • create

    Create an S3TransferManager using the default values.

    The type of S3AsyncClient used depends on if AWS Common Runtime (CRT) library software.amazon.awssdk.crt:aws-crt is in the classpath. If AWS CRT is available, an AWS CRT-based S3 client will be created via (S3AsyncClient.crtCreate()). Otherwise, a standard S3 client(S3AsyncClient.create()) will be created. Note that only AWS CRT-based S3 client supports parallel transfer, i.e, leveraging multipart upload/download for now, so it's recommended to add AWS CRT as a dependency.

  • builder

ncG1vNJzZmirlKB7ornAs6anmaeoe6S7zGihmq6RZK6xtY6lmK2do6l8tLvFra6aqpVkrq6t2ailaJmnqMClt46tqZqmo5uys3vSbGaMa4Snrq%2B%2FxZ6phpmelrSmvo2hq6ak