[Integration][Github] Add Support to Only Ingest Closed Pull Request Within a Specified Date Range (#1948)

### **User description**
# Description

What - Adds optional maxResults config option to include closed PRs
during export

Why - Users need visibility into recently closed PRs for tracking and
analysis, but ingesting all historical closed PRs would be impractical
and slow.

How - 
- Changed state selector on Pull Request to states to accept both open
and closed states
- Added maxResults config option to include closed PRs during export
- Added Batch limiting (max 100 closed PRs) to prevent performance
issues
- Modified Webhook processor to update (not delete) closed PRs when
maxResults flag is enabled pull request ingestion to GitHub integration
with time-based filtering.

## Type of change

Please leave one option from the following and delete the rest:

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] New Integration (non-breaking change which adds a new integration)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Non-breaking change (fix of existing functionality that will not
change current behavior)
- [ ] Documentation (added/updated documentation)

<h4> All tests should be run against the port production
environment(using a testing org). </h4>

### Core testing checklist

- [ ] Integration able to create all default resources from scratch
- [ ] Resync finishes successfully
- [ ] Resync able to create entities
- [ ] Resync able to update entities
- [ ] Resync able to detect and delete entities
- [ ] Scheduled resync able to abort existing resync and start a new one
- [ ] Tested with at least 2 integrations from scratch
- [ ] Tested with Kafka and Polling event listeners
- [ ] Tested deletion of entities that don't pass the selector


### Integration testing checklist

- [ ] Integration able to create all default resources from scratch
- [ ] Resync able to create entities
- [ ] Resync able to update entities
- [ ] Resync able to detect and delete entities
- [ ] Resync finishes successfully
- [ ] If new resource kind is added or updated in the integration, add
example raw data, mapping and expected result to the `examples` folder
in the integration directory.
- [ ] If resource kind is updated, run the integration with the example
data and check if the expected result is achieved
- [ ] If new resource kind is added or updated, validate that
live-events for that resource are working as expected
- [ ] Docs PR link [here](#)

### Preflight checklist

- [ ] Handled rate limiting
- [ ] Handled pagination
- [ ] Implemented the code in async
- [ ] Support Multi account

## Screenshots

Include screenshots from your environment showing how the resources of
the integration will look.

## API Documentation

Provide links to the API documentation used for this integration.


___

### **PR Type**
Enhancement


___

### **Description**
- Add optional closed pull request ingestion with time filtering

- Limit closed PRs to 60-day window and 100 per repository

- Update webhook processor to preserve closed PRs when enabled

- Maintain backward compatibility with existing configurations


___

### Diagram Walkthrough


```mermaid
flowchart LR
  A["Config Flag"] --> B["Fetch Open PRs"]
  A --> C["Fetch Closed PRs"]
  C --> D["Filter by 60 days"]
  D --> E["Limit to 100 PRs"]
  F["Webhook Event"] --> G["Check Config"]
  G --> H["Update vs Delete"]
```



<details> <summary><h3> File Walkthrough</h3></summary>

<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>4
files</summary><table>
<tr>
<td><strong>pull_request_exporter.py</strong><dd><code>Add closed PR
fetching with time filtering</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-4fa157a9e3d755d837f15ff03fc0c67a7b822e04188ffd5efba92ea6a2aca7c9">+86/-3</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>options.py</strong><dd><code>Add include_closed option to
ListPullRequestOptions</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-c47b92fb4fc063ecebd6e211ac6c6b269ad22b8e2464fe37ccd838cec4e55e79">+1/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>pull_request_webhook_processor.py</strong><dd><code>Update
webhook to preserve closed PRs conditionally</code>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-b0dd226e62e15b9cad433cc962bfab31cbc8074ce147708cefa51629753e4b8b">+5/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>main.py</strong><dd><code>Pass include_closed flag to
exporter</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-87e9377eb3998df79e7b005e93973b0b888f8431d81b49504bae4264fc148a0a">+1/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>integration.py</strong><dd><code>Add closedPullRequests
configuration field</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-352618668ce736511a5de564f4e299286cc93f0445a9f0300368ba6c93452eef">+5/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

</table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>2
files</summary><table>
<tr>
<td><strong>test_pull_request_exporter.py</strong><dd><code>Add
comprehensive tests for closed PR functionality</code>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-8d5e404f1fd0f2ee953732fc2064032bd99c768058685fb7ea669146160d3705">+214/-5</a>&nbsp;
</td>

</tr>

<tr>

<td><strong>test_pull_request_webhook_processor.py</strong><dd><code>Add
tests for webhook closed PR handling</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-3857aef3d467cc3605e6de42bbe2016de5acc83a5fbb297291e77d86ea7d04b0">+184/-1</a>&nbsp;
</td>

</tr>

</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>CHANGELOG.md</strong><dd><code>Document new closed PR
features</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-092e2c8d25f45c5b95e709fe6ab651954f49369b5616cafe9dfbdb2ec0819147">+10/-0</a>&nbsp;
&nbsp; </td>

</tr>

</table></details></td></tr><tr><td><strong>Miscellaneous</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>pyproject.toml</strong><dd><code>Bump version to
1.2.1-beta</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/port-labs/ocean/pull/1948/files#diff-d1a301fa45068b9aa15404141d1303141fb46e768af1e5a05513f414faca0299">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr></tr></tbody></table>

</details>

___

---------

Co-authored-by: Michael Kofi Armah <mikeyarmah@gmail.com>

Author: emekanwaoma

integrations/github/CHANGELOG.md

@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 <!-- towncrier release notes start -->
 
+## 1.3.0-beta (2025-08-18)
+
+
+### Improvements
+
+- Added maxResults and since config options to include closed PRs during export
+- Added Batch limiting (max 100 closed PRs) to prevent performance issues
+- Modified Webhook processor to update (not delete) closed PRs when maxResults flag is enabled
+
+
 ## 1.2.11-beta (2025-08-18)
 
 

integrations/github/github/core/exporters/pull_request_exporter.py

@@ -1,3 +1,5 @@
+from datetime import UTC, datetime, timedelta
+from typing import Any
 from github.helpers.utils import enrich_with_repository, extract_repo_params
 from port_ocean.core.ocean_types import ASYNC_GENERATOR_RESYNC_TYPE, RAW_ITEM
 from loguru import logger
@@ -26,15 +28,94 @@ async def get_paginated_resources[
     ](self, options: ExporterOptionsT) -> ASYNC_GENERATOR_RESYNC_TYPE:
         """Get all pull requests in the organization's repositories with pagination."""
 
-        repo_name, params = extract_repo_params(dict(options))
+        repo_name, extras = extract_repo_params(dict(options))
+        states = extras["states"]
+        max_results = extras["max_results"]
+        since = extras["since"]
+
+        logger.info(f"Starting pull request export for repository {repo_name}")
+
+        if "open" in states:
+            async for open_batch in self._fetch_open_pull_requests(
+                repo_name, {"state": "open"}
+            ):
+                yield open_batch
 
-        endpoint = (
+        if "closed" in states:
+            async for closed_batch in self._fetch_closed_pull_requests(
+                repo_name, max_results, since
+            ):
+                yield closed_batch
+
+    def _build_pull_request_paginated_endpoint(self, repo_name: str) -> str:
+        return (
             f"{self.client.base_url}/repos/{self.client.organization}/{repo_name}/pulls"
         )
 
+    async def _fetch_open_pull_requests(
+        self, repo_name: str, params: dict[str, Any]
+    ) -> ASYNC_GENERATOR_RESYNC_TYPE:
+        endpoint = self._build_pull_request_paginated_endpoint(repo_name)
+
         async for pull_requests in self.client.send_paginated_request(endpoint, params):
             logger.info(
-                f"Fetched batch of {len(pull_requests)} pull requests from repository {repo_name}"
+                f"Fetched batch of {len(pull_requests)} open pull requests from repository {repo_name}"
             )
             batch = [enrich_with_repository(pr, repo_name) for pr in pull_requests]
             yield batch
+
+    async def _fetch_closed_pull_requests(
+        self, repo_name: str, max_results: int, since: int
+    ) -> ASYNC_GENERATOR_RESYNC_TYPE:
+        endpoint = self._build_pull_request_paginated_endpoint(repo_name)
+        params = {
+            "state": "closed",
+            "sort": "updated",
+            "direction": "desc",
+        }
+
+        total_count = 0
+        logger.info(
+            f"[Closed PRs] Starting fetch for closed pull requests of repository {repo_name} "
+            f"with max_results={max_results}"
+        )
+
+        async for pull_requests in self.client.send_paginated_request(endpoint, params):
+            if not pull_requests:
+                logger.info(
+                    f"[Closed PRs] No more closed pull requests returned for repository {repo_name}; stopping."
+                )
+                break
+
+            remaining = max_results - total_count
+            if remaining <= 0:
+                break
+
+            # Trim batch if it would exceed max_results
+            limited_batch = pull_requests[:remaining]
+            batch_count = len(limited_batch)
+
+            logger.info(
+                f"[Closed PRs] Fetched closed pull requests batch of {batch_count} from {repo_name} "
+                f"(total so far: {total_count + batch_count}/{max_results})"
+            )
+
+            yield [
+                enrich_with_repository(pr, repo_name)
+                for pr in self._filter_prs_by_updated_at(limited_batch, since)
+            ]
+            total_count += batch_count
+
+    def _filter_prs_by_updated_at(
+        self, prs: list[dict[str, Any]], since: int
+    ) -> list[dict[str, Any]]:
+        cutoff = datetime.now(UTC) - timedelta(days=since)
+
+        return [
+            pr
+            for pr in prs
+            if datetime.strptime(pr["updated_at"], "%Y-%m-%dT%H:%M:%SZ").replace(
+                tzinfo=UTC
+            )
+            >= cutoff
+        ]

integrations/github/github/core/options.py

@@ -37,7 +37,9 @@ class SinglePullRequestOptions(RepositoryIdentifier):
 class ListPullRequestOptions(RepositoryIdentifier):
     """Options for listing pull requests."""
 
-    state: Required[str]
+    states: Required[list[str]]
+    max_results: Required[int]
+    since: Required[int]
 
 
 class SingleIssueOptions(RepositoryIdentifier):

integrations/github/github/webhook/webhook_processors/pull_request_webhook_processor.py

@@ -42,7 +42,7 @@ async def handle_event(
         logger.info(f"Processing pull request event: {action} for {repo_name}/{number}")
 
         config = cast(GithubPullRequestConfig, resource_config)
-        if action == "closed" and config.selector.state == "open":
+        if action == "closed" and "closed" not in config.selector.states:
             logger.info(
                 f"Pull request {repo_name}/{number} was closed and will be deleted"
             )

integrations/github/integration.py

@@ -70,9 +70,22 @@ class GithubFolderResourceConfig(ResourceConfig):
 
 
 class GithubPullRequestSelector(Selector):
-    state: Literal["open", "closed", "all"] = Field(
-        default="open",
-        description="Filter by pull request state (e.g., open, closed, all)",
+    states: list[Literal["open", "closed"]] = Field(
+        default=["open"],
+        description="Filter by pull request state (e.g., open, closed)",
+    )
+    max_results: int = Field(
+        alias="maxResults",
+        default=100,
+        ge=1,
+        le=300,
+        description="Limit the number of pull requests returned",
+    )
+    since: int = Field(
+        default=60,
+        ge=1,
+        le=90,
+        description="Only fetch pull requests created within the last N days (1-90 days)",
     )
 
 

integrations/github/main.py

@@ -235,7 +235,9 @@ async def resync_pull_requests(kind: str) -> ASYNC_GENERATOR_RESYNC_TYPE:
             pull_request_exporter.get_paginated_resources(
                 ListPullRequestOptions(
                     repo_name=repo["name"],
-                    state=config.selector.state,
+                    states=list(config.selector.states),
+                    max_results=config.selector.max_results,
+                    since=config.selector.since,
                 )
             )
             for repo in repos

integrations/github/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "github-ocean"
-version = "1.2.11-beta"
+version = "1.3.0-beta"
 description = "This integration ingest data from github"
 authors = ["Chukwuemeka Nwaoma <joelchukks@gmail.com>", "Melody Anyaegbulam <melodyogonna@gmail.com>", "Michael Armah <mikeyarmah@gmail.com>"]