You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+79-2Lines changed: 79 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,9 +10,86 @@ This is an experimental polite downloader for Common Crawl data written in `rust
10
10
11
11
## Installation
12
12
13
-
For now, the only supported way to install the tool is to use `cargo`. For this you need to have `rust` installed. You can install `rust` by following the instructions on the [official website](https://www.rust-lang.org/tools/install).
13
+
You can install `cc-downloader`` via our pre-built binaries, or by compiling it from source.
14
14
15
-
After installing `rust`, ``cc-downloader`` can be installed with the following command:
15
+
### Pre-built binaries
16
+
17
+
You can find our pre-built binaries on our [GitHub releases page](https://github.com/commoncrawl/cc-downloader/releases). They are available for `Linux`, `macOS`, and `Windows`, in `x86_64` and `aarch64` architectures (Windows is only supported in `x86_64`). In order to use them please select and download the correct binary for your system.
After downloading it, please verify the checksum of the binary. You can find the checksum file in the same location as the binary. The checksum is generated using `sha512sum`. You can verify it by running the following command:
If the checksum is valid, which will be indicated by and `OK` message, you can proceed to extract the binary. For `tar.gz` files you can use the following command:
31
+
32
+
```bash
33
+
tar -xzf cc-downloader-[VERSION]-[ARCH]-[OS].tar.gz
34
+
```
35
+
36
+
For `zip` files you can use the following command:
37
+
38
+
```bash
39
+
unzip cc-downloader-[VERSION]-[ARCH]-[OS].zip
40
+
```
41
+
42
+
This will extract the binary, the licenses and the readme file **in the current folder**. After extracting the binary, you can run it by executing the following command:
43
+
44
+
```bash
45
+
./cc-downloader
46
+
```
47
+
48
+
If you want to use the binary from anywhere, you can move it to a folder in your `PATH`. For more information on how to do this, please refer to the documentation of your operating system. For example, on `Linux` and `macOS` you can move it to `~/.bin`:
49
+
50
+
```bash
51
+
mv cc-downloader ~/.bin
52
+
```
53
+
54
+
And then add the following line to your `~/.bashrc` or `~/.zshrc` file:
55
+
56
+
```bash
57
+
export PATH=$PATH:~/.bin
58
+
```
59
+
60
+
then run the following command to apply the changes:
61
+
62
+
```bash
63
+
source~/.bashrc
64
+
```
65
+
66
+
or
67
+
68
+
```bash
69
+
source~/.zshrc
70
+
```
71
+
72
+
Then, you can run the binary from anywhere. If you want to update the binary, you can repeat the process and download the new version. Make sure to replace the binary that is stored in the folder that you added to your `PATH`. If you want to remove the binary, you can simply delete from this folder.
73
+
74
+
### Compiling from source
75
+
76
+
For this you need to have `rust` installed. You can install `rust` by following the instructions on the [official website](https://www.rust-lang.org/tools/install).
77
+
78
+
Or by running the following command:
79
+
80
+
```bash
81
+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
82
+
```
83
+
84
+
Even if you have `rust` a system-wide installation, we recommend the linked installation method. A system-wide installation and a user installation can co-exist without any problems.
85
+
86
+
When compiling from source, please make sure you have the latest version of `rust` installed by running the following command:
87
+
88
+
```bash
89
+
rustup update
90
+
```
91
+
92
+
Now you can install the `cc-downloader` tool by running the following command:
0 commit comments