Add Integration Tests #816
+1,315
−21
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| name: Integration Tests | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: | ||
| - master | ||
| - production | ||
| types: | ||
| - opened | ||
| - synchronize | ||
| - reopened | ||
|
|
||
| jobs: | ||
| integration-tests: | ||
| name: Run Integration Tests | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 15 | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0 | ||
|
|
||
| - name: Run integration tests | ||
| run: make integration-test | ||
|
|
||
| - name: Cleanup | ||
| if: always() | ||
| run: make integration-test-clean | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| FROM rust:1.69 AS rust_builder | ||
| RUN rustup target add x86_64-unknown-linux-musl | ||
| RUN apt-get update && apt-get install -y musl-tools | ||
| RUN git clone https://github.com/brave-intl/challenge-bypass-ristretto-ffi /src | ||
| WORKDIR /src | ||
| RUN git checkout 1.0.1 | ||
| RUN CARGO_PROFILE_RELEASE_LTO=true cargo rustc --target=x86_64-unknown-linux-musl --release --crate-type staticlib | ||
|
|
||
| FROM golang:1.24 | ||
| WORKDIR /app | ||
| COPY . . | ||
|
|
||
| # Copy the Rust FFI library from the rust builder stage | ||
| COPY --from=rust_builder /src/target/x86_64-unknown-linux-musl/release/libchallenge_bypass_ristretto_ffi.a /usr/lib/libchallenge_bypass_ristretto_ffi.a | ||
|
|
||
| RUN apt-get update && apt-get install -y postgresql-client | ||
| RUN go mod download | ||
| RUN go get github.com/stretchr/testify/assert | ||
| RUN go get github.com/segmentio/kafka-go | ||
|
|
||
| CMD ["go", "test", "-tags", "integration", "./integration-tests", "-v"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,12 +21,79 @@ This project uses [golangci-lint](https://golangci-lint.run/) for linting, this | |
| To run locally use `make lint` which runs linting using docker however if you want to run it locally using a binary release (which can be faster) follow the [installation instructions for your platform](https://golangci-lint.run/usage/install/) and then run `golangci-lint run -v ./...` | ||
|
|
||
| ## Testing | ||
|
|
||
| ### Unit Tests | ||
|
|
||
| Run the below command in order to test changes, if you have an M1 / M2 Mac (or ARM based processor) follow the steps below to setup docker to be able to run the tests | ||
| ``` | ||
| make docker-test | ||
| ``` | ||
|
|
||
| ### Integration Tests | ||
|
|
||
| The project includes comprehensive integration tests that verify the entire system working together with all dependencies. | ||
|
|
||
| #### What the Integration Tests Do | ||
|
|
||
| The integration tests: | ||
| - Spin up a complete environment with PostgreSQL, Kafka, Zookeeper, LocalStack (for DynamoDB), and the application | ||
| - Test end-to-end flows including: | ||
| - Token redemption flows through Kafka | ||
| - Token signing flows through Kafka | ||
| - Database persistence and retrieval | ||
| - DynamoDB operations | ||
| - Verify the application correctly processes messages between Kafka topics | ||
| - Ensure proper communication between all services | ||
|
|
||
| #### Running Integration Tests | ||
|
There was a problem hiding this comment. question: is there a way to run a single integration test? There was a problem hiding this comment. I'll add documentation about the There was a problem hiding this comment. To run that we need to bring up the containers and test environment so we would need to run make dev then from the shell run the It would be good to have something where we can run a single test and pick up and code changes we are making locally, similar to bat-go. There was a problem hiding this comment. I see your meaning. Good point will add. |
||
|
|
||
| To run the integration tests, simply use: | ||
|
|
||
| ```bash | ||
| # run all integration tests | ||
| make integration-test | ||
| # or run a specific integration test | ||
| make integration-test TEST_NAME=TestTokenIssuanceViaKafkaAndRedeemViaHTTPFlow | ||
| ``` | ||
|
|
||
| This command will: | ||
| 1. Clean up any existing test containers | ||
| 2. Build all required services | ||
| 3. Start the test environment (PostgreSQL, Kafka, Zookeeper, LocalStack) | ||
| 4. Wait for all services to be healthy and ready (~30 seconds) | ||
| 5. Build and run the test suite | ||
| 6. Automatically clean up all containers and volumes after completion | ||
|
|
||
| #### Manual Cleanup | ||
|
|
||
| If the tests are interrupted or you need to manually clean up the test environment: | ||
|
|
||
| ```bash | ||
| make integration-test-clean | ||
| ``` | ||
|
|
||
| This will remove all test containers, networks, and volumes created by the integration tests. | ||
|
|
||
| #### Viewing Logs | ||
|
|
||
| To debug issues or view what's happening during the tests: | ||
|
|
||
| ```bash | ||
| make integration-test-logs | ||
| ``` | ||
|
|
||
| This will tail the logs from all services in the integration test environment. | ||
|
|
||
| #### Test Configuration | ||
|
|
||
| The integration tests use a separate `docker-compose.integration.yml` file which: | ||
| - Creates isolated test topics in Kafka | ||
| - Uses a dedicated test database | ||
| - Runs LocalStack for DynamoDB emulation | ||
| - Configures all services with test-specific settings | ||
|
|
||
| ### Have an M1 / M2 (ARM) Mac? | ||
|
|
||
| 1.) In Docker Desktop, go to: `Settings -> Docker Engine` <br /> | ||
| #### Modify file to include | ||
| ``` | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| services: | ||
| cbp: | ||
| build: . | ||
| depends_on: | ||
| postgres: | ||
| condition: service_healthy | ||
| kafka: | ||
| condition: service_healthy | ||
| localstack: | ||
| condition: service_healthy | ||
| environment: | ||
| - DATABASE_URL=postgresql://testuser:testpassword@postgres:5432/testdb?sslmode=disable | ||
Sneagan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - VPC_KAFKA_BROKERS=kafka:9092 | ||
| - DYNAMODB_ENDPOINT=http://localstack:4566 | ||
| - ENV=test | ||
| - REDEEM_CONSUMER_TOPIC=test.consumer.redeem | ||
| - REDEEM_PRODUCER_TOPIC=test.producer.redeem | ||
| - SIGN_CONSUMER_TOPIC=test.consumer.sign | ||
| - SIGN_PRODUCER_TOPIC=test.producer.sign | ||
| - CONSUMER_GROUP=integration-service | ||
| - AWS_REGION=us-east-1 | ||
| - AWS_ACCESS_KEY_ID=test | ||
| - AWS_SECRET_ACCESS_KEY=test | ||
| ports: | ||
| - "2416:2416" | ||
| - "9090:9090" | ||
|
|
||
| postgres: | ||
| image: postgres:latest | ||
| environment: | ||
| - POSTGRES_USER=testuser | ||
| - POSTGRES_PASSWORD=testpassword | ||
| - POSTGRES_DB=testdb | ||
| ports: | ||
| - "5432:5432" | ||
| healthcheck: | ||
| test: ["CMD-SHELL", "pg_isready -U testuser -d testdb"] | ||
| interval: 5s | ||
| timeout: 2s | ||
| retries: 10 | ||
|
|
||
| zookeeper: | ||
| image: wurstmeister/zookeeper:latest | ||
| ports: | ||
| - "2181:2181" | ||
| healthcheck: | ||
| test: ["CMD-SHELL", "echo ruok | nc localhost 2181 | grep imok"] | ||
| interval: 5s | ||
| timeout: 2s | ||
| retries: 10 | ||
|
|
||
| kafka: | ||
| image: wurstmeister/kafka:latest | ||
| depends_on: | ||
| zookeeper: | ||
| condition: service_healthy | ||
| ports: | ||
| - "9092:9092" | ||
| environment: | ||
| - KAFKA_ADVERTISED_HOST_NAME=kafka | ||
| - KAFKA_ADVERTISED_PORT=9092 | ||
| - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 | ||
| - KAFKA_CREATE_TOPICS=test.consumer.redeem:1:1,test.producer.redeem:1:1,test.consumer.sign:1:1,test.producer.sign:1:1 | ||
| - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true | ||
| # Reduce memory usage for testing | ||
| - KAFKA_HEAP_OPTS=-Xmx256M -Xms128M | ||
| healthcheck: | ||
| test: ["CMD-SHELL", "kafka-topics.sh --bootstrap-server localhost:9092 --list"] | ||
| interval: 10s | ||
| timeout: 5s | ||
| retries: 10 | ||
|
|
||
| localstack: | ||
| image: localstack/localstack:latest | ||
| ports: | ||
| - "4566:4566" | ||
| environment: | ||
| - SERVICES=dynamodb | ||
| - DEBUG=1 | ||
| healthcheck: | ||
| test: ["CMD", "sh", "-c", "awslocal dynamodb list-tables --region us-east-1 || exit 1"] | ||
| interval: 5s | ||
| timeout: 5s | ||
| retries: 15 | ||
|
|
||
| test-runner: | ||
| build: | ||
| context: . | ||
| dockerfile: Dockerfile.test | ||
| profiles: | ||
| - test | ||
| depends_on: | ||
| cbp: | ||
| condition: service_started | ||
| kafka: | ||
| condition: service_healthy | ||
| environment: | ||
| - DATABASE_URL=postgresql://testuser:testpassword@postgres:5432/testdb?sslmode=disable | ||
Sneagan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - DYNAMODB_ENDPOINT=http://localstack:4566 | ||
| - ENV=test | ||
| - TEST_SHOULD_WRITE_REDEEM_REQUESTS_HERE=test.consumer.redeem | ||
| - TEST_SHOULD_READ_REDEEM_REQUESTS_HERE=test.producer.redeem | ||
| - TEST_SHOULD_WRITE_SIGNING_REQUESTS_HERE=test.consumer.sign | ||
| - TEST_SHOULD_READ_SIGNING_REQUESTS_HERE=test.producer.sign | ||
| - CONSUMER_GROUP=test | ||
| command: ["go", "test", "-tags", "integration", "./integration-tests", "-v", "-count=1"] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.