Skip to content

feature: add extra virtual host style endpoints support #795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

LinPr
Copy link

@LinPr LinPr commented Feb 18, 2025

This PR add the user custom endpoints that need to send request with virtual host style

reference issues:
#794
#518

Here I add a command line flag and also support fetch config from environment variable,

--addressing-style value use virtual host style or path style endpoint: (path, virtual) (default: path)

the address-style config is compatible with the aws s3 cli config,here is the aws document , and you can just search the address_style key word in the page. https://docs.aws.amazon.com/cli/latest/topic/s3-config.html

NAME:
   s5cmd - Blazing fast S3 and local filesystem execution tool

USAGE:
   s5cmd [global options] command [command options] [arguments...]

COMMANDS:
   ls              list buckets and objects
   cp              copy objects
   rm              remove objects
   mv              move/rename objects
   mb              make bucket
   rb              remove bucket
   select          run SQL queries on objects
   du              show object size usage
   cat             print remote object content
   pipe            stream to remote from stdin
   run             run commands in batch
   sync            sync objects
   version         print version
   bucket-version  configure bucket versioning
   presign         print remote object presign url
   head            print remote object metadata
   help, h         Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --addressing-style value       use virtual host style or path style endpoint: (path, virtual) (default: path) [$S3_ADDRESSING_STYLE]
   --credentials-file value       use the specified credentials file instead of the default credentials file
   --dry-run                      fake run; show what commands will be executed without actually executing them (default: false)
   --endpoint-url value           override default S3 host for custom services [$S3_ENDPOINT_URL]
   --help, -h                     show help (default: false)
   --install-completion           get completion installation instructions for your shell (only available for bash, pwsh, and zsh) (default: false)
   --json                         enable JSON formatted output (default: false)
   --log value                    log level: (trace, debug, info, error) (default: info)
   --no-sign-request              do not sign requests: credentials will not be loaded if --no-sign-request is provided (default: false)
   --no-verify-ssl                disable SSL certificate verification (default: false)
   --numworkers value             number of workers execute operation on each object (default: 256)
   --profile value                use the specified profile from the credentials file
   --request-payer value          who pays for request (access requester pays buckets)
   --retry-count value, -r value  number of times that a request will be retried for failures (default: 10)
   --stat                         collect statistics of program execution and display it at the end (default: false)
   --use-list-objects-v1          use ListObjectsV1 API for services that don't support ListObjectsV2 (default: false)

And I well tested it, using this on our daily work, no bug being found till now.

I can give a example here :

  1. If I want to upload a file to remote object storage , Firstly, I need to set up config with environment variable
export AWS_ACCESS_KEY_ID='xxxxxxxxxxx'
export AWS_SECRET_ACCESS_KEY='xxxxxxxxxx'
export S3_ENDPOINT_URL='http://oss-cn-hangzhou.aliyuncs.com'
  1. Secondly, because I want to sent request with virtual host style, I need a another env or command line flag
export S3_ADDRESSING_STYLE="virtual"      
  1. Now you can upload or down load file with the s5cmd command
/s5cmd cp ./testfile s3://restic-backup-good/hello/
or I can run:
/s5cmd  --addressing-style=virtual cp ./testfile s3://restic-backup-good/hello/

output:
cp ./testfile s3://restic-backup-good/hello/testfile 

let’s see the request detail on wireshark. it just send the request with virtual host style , a bucket name as prefix of host

image

@LinPr LinPr requested a review from a team as a code owner February 18, 2025 04:16
@LinPr LinPr requested review from ilkinulas and sonmezonur and removed request for a team February 18, 2025 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant