PredPatt Integration and Python 3.12+ Modernization

[aaronstevenwhite]

This PR represents a significant modernization of the Decomp toolkit with full integration of PredPatt predicate-argument structure extraction functionality and comprehensive Python 3.12+ compatibility updates.

Summary

This PR integrates the standalone PredPatt library directly into decomp as decomp.semantics.predpatt, modernizes the codebase for Python 3.12+, and adds comprehensive CI/CD infrastructure. The integration maintains complete compatibility with the original PredPatt implementation while providing seamless interoperability with the UDS framework.

Key Changes

1. PredPatt Integration (~7,000 lines)

• Full module structure at decomp.semantics.predpatt:
  • core/: Core data structures (Token, Predicate, Argument, PredPattOpts)
  • extraction/: Main extraction engine with linguistic rule application
  • parsing/: Universal Dependencies parsing utilities
  • rules/: Modular linguistic rules for predicate/argument identification
  • filters/: Configurable filtering system
  • utils/: Visualization and debugging utilities
• Algorithm fidelity: Byte-for-byte identical output with standalone PredPatt v1.0.1
• Comprehensive testing: 100+ differential tests ensure compatibility

2. Python 3.12+ Modernization

• Type system updates:
  • Modern union syntax (X | Y instead of Union[X, Y])
  • Built-in generics (list[str] instead of List[str])
  • type aliases instead of TypeAlias
  • Protocol-based typing for better type safety
• Packaging modernization:
  • Migration from setup.py to pyproject.toml
  • Structured dependency management with optional [dev] extras
  • Modern build system configuration

3. CI/CD Infrastructure

• GitHub Actions workflow (.github/workflows/ci.yml):
  • Automated testing with pytest (including slow tests)
  • Linting with ruff (E/F errors only, style warnings suppressed)
  • Type checking with mypy
  • Documentation building with warnings as errors
• Development tooling:
  • ruff.toml: Linting and formatting configuration
  • mypy.ini: Type checking configuration
  • Comprehensive test suite with 400+ tests

4. Documentation Enhancements

• Comprehensive API documentation:
  • Detailed module and class docstrings throughout
  • Structured documentation for PredPatt components
  • Type hints for all public APIs
• Updated tutorials:
  • Installation instructions for modern packaging
  • Quick-start guide corrections
  • Docker setup with jupyter/datascience-notebook
• New content:
  • CHANGELOG.md with complete release history
  • releases.rst documentation page
  • CI/CD documentation

5. Bug Fixes

• Fixed JSON serialization issue in UDS metadata (sets → lists)
• Corrected document_ids → documentids in documentation
• Fixed trailing whitespace in linearization docstrings
• Resolved F841 unused variable warnings

Testing

All tests pass including the comprehensive PredPatt differential test suite:

pytest --runslow  # 417 tests pass
mypy decomp      # No type errors
ruff check . --select E,F  # No critical errors

Breaking Changes

None for existing users. The integration is additive:

• Original decomp API remains unchanged
• PredPatt functionality is now available at decomp.semantics.predpatt
• Python 3.12+ is now required (was 3.6+)

Migration Guide

For users of standalone PredPatt:

# before (standalone predpatt)
from predpatt import PredPatt, load_conllu, PredPattOpts

# after (integrated in decomp)
from decomp.semantics.predpatt import PredPatt, load_conllu, PredPattOpts

Example Usage

from decomp.semantics.predpatt import PredPatt, load_conllu

# load dependency parse
sentences = load_conllu('example.conllu')

# extract predicates and arguments
pp = PredPatt(sentences[0])

# access structures
for predicate in pp.predicates:
    print(f"Predicate: {predicate}")
    for arg in predicate.arguments:
        print(f"  Argument: {arg}")

Future Work

• Standalone PredPatt contains a syntactic parser. This parser is very old and implemented in java. A newer parser will be integrated in the future under a parsing feature. Currently this feature has concrete as a dependency, but this dependency is likely to be removed, since it is unlikely to be necessary and is requires many old dependencies.
• The style of the code is relatively homogeneous but could be improved.
• Some unpythonic naming conventions from the standalone PredPatt remain for the sake of backward compatibility and should be removed.
• The tests need substantial reorganization, style edits, and deduplication, especially the differential tests comparing standalone PredPatt to the integrated implementation.

Checklist

• Tests pass locally and in CI
• Documentation builds without warnings
• Type checking passes
• Linting passes (critical errors only)
• Changelog updated
• README updated with new badges and instructions