FetchFiles

A command-line tool for downloading files in parallel from a JSON configuration file.

FetchFiles is a Go-based command-line tool that allows you to download multiple files in parallel from a JSON configuration file. It provides a simple and efficient way to fetch files from various sources and save them to your local machine.

Installing / Getting started

To use FetchFiles, you'll need to have Go installed on your system. You can then install the tool using the following command:

$ go install github.com/walker84837/fetchfiles@latest

Once installed, you can run the tool with the following command:

$ fetchfiles -batch path/to/batch.json

This will download the files specified in the batch.json file in parallel, using the number of concurrent downloads specified by the -concurrency flag (default is 2).

Initial Configuration for New Users

To get started with FetchFiles, all you need is a simple JSON file that tells the program which files to download and where to place them. Here's an example of what this file might look like:

{
    "file1.txt": "https://example.com/file1.txt",
    "file2.zip": "https://example.com/file2.zip",
    "directory/file3.pdf": "https://example.com/file3.pdf"
}

In this JSON file, the keys represent the local paths where you want the downloaded files to be saved, and the values are the URLs from which the files will be downloaded.

To make things even easier, we've provided a collection of preset JSON batch files in the presets folder. You can use these as a starting point, modify them to suit your needs, or even contribute your own if you'd like.

Developing

If you'd like to contribute to the development of FetchFiles, you can follow these steps:

$ git clone https://github.com/walker84837/fetchfiles.git
$ cd fetchfiles
$ go build

This will clone the repository, build the project, and create an executable file that you can run.

Building

To build the FetchFiles binary, simply run the following command:

$ go build

This will create an executable file named fetchfiles in the current directory.

Features

• Parallel file downloads using goroutines and a semaphore for concurrency control
• Verbose logging for detailed progress information
• Customizable number of concurrent downloads

Configuration

FetchFiles supports the following command-line arguments:

Flag	Type	Default	Description
-batch	string	Mandatory	The path to the JSON file that contains the list of files to be downloaded. Check Initial Configuration for more information.
-concurrency	int	2	The number of concurrent downloads to perform.
-verbose	bool	false	Enables verbose logging, which provides more detailed information about the download progress.

Contributing

If you'd like to contribute to FetchFiles, please fork the repository and use a feature branch. Pull requests are warmly welcome.

Links

• Project homepage: https://github.com/walker84837/fetchfiles
• Repository: https://github.com/walker84837/fetchfiles
• Issue tracker: https://github.com/walker84837/fetchfiles/issues

Licensing

The code in this project is licensed under the BSD 3-Clause license. The logo is licensed under the CC BY-NC-SA 4.0.