chore: Merge pull request #7 from commoncrawl/dev

chore: Merges fixes, features and refactors for version 0.5.2

Fixes issue feat: Add a User Agent to cc-downloader #6
- Introduces refactors so that linter check are all passed
- Introduces a rust workflow for ensuring that the code compiles and test are passed in the dev and main branches
- Introduces changes to the contributing policy so that PRs are merged to the dev branch
- Introduces slight updates to the documentation

Author: pjox

.github/workflows/rust.yml

@@ -0,0 +1,25 @@
+name: Rust
+
+on:
+  push:
+    branches: [ "main", "dev" ]
+  pull_request:
+    branches: [ "main", "dev" ]
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build
+      run: cargo build --verbose
+    - name: Run tests
+      run: cargo test --verbose
+    - name: Run clippy
+      run: cargo clippy --verbose
+      

CONTRIBUTING.md

@@ -1,4 +1,5 @@
 # How to contribute to cc-downloader?
+
 [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](CODE_OF_CONDUCT.md)
 
 `cc-downloader` is an open source project, so all contributions and suggestions are welcome.
@@ -12,6 +13,7 @@ In order to facilitate healthy, constructive behavior in an open and inclusive c
 our [code of conduct](CODE_OF_CONDUCT.md).
 
 ## How to work on an open Issue?
+
 You have the list of open Issues at: [https://github.com/commoncrawl/cc-downloader/issues](https://github.com/commoncrawl/cc-downloader/issues)
 
 Some of them may have the label `help wanted`: that means that any contributor is welcomed!
@@ -36,13 +38,14 @@ If you would like to work on any of the open Issues:
     git remote add upstream git@github.com:commoncrawl/cc-downloader.git
     ```
 
-3. Create a new branch to hold your development changes:
+3. Switch to the `dev` branch and then create a new branch to hold your development changes:
 
     ```bash
+    git checkout dev
     git checkout -b a-descriptive-name-for-my-changes
     ```
 
-    **do not** work on the `main` branch.
+    **do not** work on the `main` or `dev` branches.
 
 4. Develop the features on your branch.
 
@@ -58,17 +61,16 @@ If you would like to work on any of the open Issues:
 
     ```bash
     git fetch upstream
-    git rebase upstream/main
+    git rebase upstream/dev
     ```
 
-9. Once you are satisfied, push the changes to your fork repo using:
+6. Once you are satisfied, push the changes to your fork repo using:
 
    ```bash
    git push -u origin a-descriptive-name-for-my-changes
    ```
 
-   Go the webpage of your fork on GitHub. Click on "Pull request" to send your to the project maintainers for review.
-
+   Go the webpage of your fork on GitHub. Click on "Pull request" to send your to the project maintainers for review, and select the `dev` branch as the brach you'd like to merge your changes into.
 
 Thank you for your contribution!
 

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "cc-downloader"
-version = "0.5.1"
+version = "0.5.2"
 edition = "2021"
 authors = ["Pedro Ortiz Suarez <pedro@commoncrawl.org>"]
 description = "A polite and user-friendly downloader for Common Crawl data."
@@ -12,16 +12,24 @@ repository = "https://github.com/commoncrawl/cc-downloader"
 documentation = "https://docs.rs/cc-downloader"
 
 [dependencies]
-clap = { version = "4.5.23", features = ["derive"] }
+clap = { version = "4.5.29", features = ["derive"] }
 flate2 = "1.0.35"
 futures = "0.3.31"
-indicatif = "0.17.9"
-reqwest = { version = "0.12.9", default-features = false, features = [
+indicatif = "0.17.11"
+reqwest = { version = "0.12.12", default-features = false, features = [
     "stream",
     "rustls-tls",
 ] }
 reqwest-middleware = "0.4.0"
 reqwest-retry = "0.7.0"
-tokio = { version = "1.42.0", features = ["full"] }
+tokio = { version = "1.43.0", features = ["full"] }
 tokio-util = { version = "0.7.13", features = ["compat"] }
 url = "2.5.4"
+
+[dev-dependencies]
+serde = { version = "1.0.217", features = ["derive"] }
+reqwest = { version = "0.12.12", default-features = false, features = [
+    "stream",
+    "rustls-tls",
+    "json",
+] }

README.md

@@ -5,7 +5,7 @@ This is an experimental polite downloader for Common Crawl data writter in `rust
 ## Todo
 
 - [ ] Add Python bindings
-- [ ] Add tests
+- [ ] Add more tests
 - [ ] Handle unrecoverable errors
 
 ## Installation

SECURITY.md

@@ -11,4 +11,4 @@ Only the latest minor version is being supported
 
 ## Reporting a Vulnerability
 
-To report a security vulnerability, please contact: info@commoncrawl.org
+To report a security vulnerability, please contact: info[at]commoncrawl[dot]org