Skip to content

Hindi TN Future Implementations 2.0. - Fraction, Measure and Time #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: staging_hi_tn
Choose a base branch
from

Conversation

ngachchi
Copy link
Contributor

@ngachchi ngachchi commented Aug 7, 2025

What does this PR do ?

This PR introduces Hindi Text Normalization 2.0, which features substantial accuracy improvements across multiple classes. It also integrates culturally relevant linguistic constructs to enhance natural language understanding.

Key Enhancements:

  • Linguistic Enrichment:
    • Incorporated quarterly terms frequently used in Hindi for Fraction, Measure, and Time classes:
      • Savva (सवा), Saadhe (साढ़े), Ponne (पौने), Dedh (डेढ़), Dhai (ढाई)

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

mgrafu and others added 2 commits April 24, 2025 11:20
* Future Implementations for classes - Measure, Money, and Date (NVIDIA#258)

* Future Implementations for classes - Measure, Money, and Date

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* Resolved the conflicts with mm_yyyy and date ranges and added the previously removed failing test cases.

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed the unused empty string implementation

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor fixes for the tagger files

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reformatted decimal final graph

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* incorporated the suggestion for decimal graph

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Century implementations

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* Working on the yyyy format for the date class

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* reverted yyyy code

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* working on future implementations

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* working on improving the date class accuracy

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added year prefix for the date class

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* working on the commma cases for date class

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* minor fixes

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* implemented mixed fractions

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* rectified the test case

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* working on quarterly measurements

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* reformatted the prefixes and suffixes for date tagger class

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* replaced text tag with era tag for the date class

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* Removed the text tag reference from date class verbalizer

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

---------

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* update jenkins cache

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Potential fix for code scanning alert no. 821: Unused local variable

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>

---------

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Co-authored-by: Namrata Gachchi <ngachchi@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
@mgrafu mgrafu changed the base branch from main to staging_hi_tn August 11, 2025 20:49
ngachchi and others added 2 commits August 12, 2025 11:32
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Copy link

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@github-actions github-actions bot added the Stale label Aug 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants