Merge pull request #179 from lanl/develop

Develop

Author: MaksimEkin

.github/workflows/ci_tests.yml

@@ -18,15 +18,17 @@ jobs:
       - uses: conda-incubator/setup-miniconda@v2
         with:
           activate-environment: anaconda-client-env
-          python-version: 3.11
+          python-version: 3.11.10
           auto-activate-base: True
       - name: Install Dependencies and T-ELF
         run: |
           conda info
           conda list
-          pip install -e .
+          pip install pytest
+          pip install poetry
+          poetry install
           
       - name: Run the Unittests
         run: |
-          cd tests && python -m pytest *
+          cd tests && poetry run python -m pytest *
       

CITATION.cff

@@ -20,7 +20,7 @@ authors:
   - family-names: Alexandrov
     given-names: Boian
 title: "Tensor Extraction of Latent Features (T-ELF)"
-version: 0.0.20
+version: 0.0.34
 url: https://github.com/lanl/T-ELF
 doi: 10.5281/zenodo.10257897
 date-released: 2023-12-04

README.md

@@ -2,7 +2,7 @@
 
 <div align="center", style="font-size: 50px">
 
-[![Build Status](https://github.com/lanl/T-ELF/actions/workflows/ci_tests.yml/badge.svg?branch=main)](https://github.com/lanl/T-ELF/actions/workflows/ci_tests.yml/badge.svg?branch=main) [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg) [![Python Version](https://img.shields.io/badge/python-v3.11.5-blue)](https://img.shields.io/badge/python-v3.8.5-blue) [![DOI](https://zenodo.org/badge/703212457.svg)](https://zenodo.org/doi/10.5281/zenodo.10257896)
+[![Build Status](https://github.com/lanl/T-ELF/actions/workflows/ci_tests.yml/badge.svg?branch=main)](https://github.com/lanl/T-ELF/actions/workflows/ci_tests.yml/badge.svg?branch=main) [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg) [![Python Version](https://img.shields.io/badge/python-v3.11.10-blue)](https://img.shields.io/badge/python-v3.11.10-blue) [![DOI](https://zenodo.org/badge/703212457.svg)](https://zenodo.org/doi/10.5281/zenodo.10257896)
 
 </div> 
 
@@ -33,59 +33,40 @@ T-ELF's adaptability spans across a multitude of disciplines, positioning it as
 
 ## Installation
 
-### Step 1: Install the Library
+### Step 1: [Install Poetry to your system](https://python-poetry.org/docs/)
+This step is optional. Use Pip or Conda if Poetry is not avaiable.
 
-**Option 1: Install via PIP**
-```shell
-conda create --name TELF python=3.11.5
-source activate TELF # or <conda activate TELF>
-pip install git+https://github.com/lanl/T-ELF.git
-```
+### Step 2: Install the Library
 
-**Option 2: Install from Source**
+**Option 1: Install via Poetry or Pip**
 ```shell
-git clone https://github.com/lanl/T-ELF.git
-cd T-ELF
-conda create --name TELF python=3.11.5
+conda create --name TELF python=3.11.10
 source activate TELF # or <conda activate TELF>
-pip install -e . # or <python setup.py install>
+poetry install # or <pip install .>
 ```
 
-**Option 3: Install via Conda**
+**Option 2: Install via Conda**
 ```shell
-git clone https://github.com/lanl/T-ELF.git
-cd T-ELF
+git clone https://gitlab.lanl.gov/maksim/telf_internal
+cd telf_internal
 conda env create --file environment_gpu.yml # use <conda env create --file environment_cpu.yml> for CPU only
 conda activate TELF_conda
 conda develop .
 ```
 
-### Step 2: Install Spacy NLP model and NLTK Packages
+### Step 3: Post-installation Dependencies
+Next, we need to install the optional and additional dependencies. These include optional dependencies for GPU and HPC capabilities, as well as required dependencies like the SpaCy language models.
+To view all available options, please run:
 ```shell
-python -m spacy download en_core_web_lg
-python -m spacy download en_core_web_trf
-python -m nltk.downloader wordnet omw-1.4
+python post_install.py --help
 ```
-
-### Step 3: Install Cupy if using GPU (*Optional* - Skip if used *Option 3* in *Step 1*)
+Install the additional dependencies:
 ```shell
-conda install -c conda-forge cupy
-```
-
-### Step 4: Install MPI if using HPC (*Optional*)
-```shell
-module load <openmpi> # On a HPC Node
-pip install mpi4py # or <conda install -c conda-forge mpi4py> depending on the system
+python post_install.py # use the following, for example, for GPU system: <python post_install.py --gpu>
 ```
 
 #### Jupyter Setup Tutorial for using the examples ([Link](https://www.maksimeren.com/post/conda-and-jupyter-setup-for-research/))
 
-### Other Considerations
-On some Linux devices, based on how CUDA was configured, you may get an error when using a GPU. Install ```cudatoolkit``` to resolve the error:
-```shell
-conda install cudatoolkit
-conda install cudnn
-```
 
 
 ## Capabilities
@@ -114,46 +95,30 @@ conda install cudnn
 |           RNMFk           | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |  :heavy_check_mark: | :heavy_check_mark: |                         Recommender NMFk                                         |   [Link](examples/RNMFk/RNMFk.ipynb)  |       :white_check_mark:       |
 |           SymNMFk         | :heavy_check_mark: |                    | :heavy_check_mark: | :heavy_check_mark: |  :heavy_check_mark: | :heavy_check_mark: |                         NMFk with Symmetric Clustering                           |   [Link](examples/SymNMFk/SymNMFk.ipynb)          |       :white_check_mark:       |
 |           WNMFk           | :heavy_check_mark: |                    | :heavy_check_mark: | :heavy_check_mark: |  :heavy_check_mark: | :heavy_check_mark: |                         NMFk with weighting - used for recommendation system     |   [Link](examples/WNMFk/WNMFk.ipynb)          |       :white_check_mark:       |
-|           HNMFk           | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |  :heavy_check_mark: | :heavy_check_mark: |                         Hierarchical NMFk                                        |      [Link](examples/HNMFk/HNMFk.ipynb)       |       :white_check_mark:       |
-|           BNMFk           |                    |                    |                    |                    |                     |                    |                           Boolean NMFk                                           |             |       :soon:       |
-|         SPLIT NMFk        |                    |                    |                    |                    |                     |                    |        Joint NMFk factorization of multiple data via SPLIT                       |             |       :soon:       |
-| SPLIT Transfer Classifier |                    |                    |                    |                    |                     |                    |      Supervised transfer learning method via SPLIT and NMFk                      |             |       :soon:       |
+|           HNMFk           | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |  :heavy_check_mark: | :heavy_check_mark: |                         Hierarchical NMFk                                        |   [Link](examples/HNMFk/HNMFk.ipynb)       |       :white_check_mark:       |
+|           BNMFk           | :heavy_check_mark: |                    | :heavy_check_mark: | :heavy_check_mark: |  :heavy_check_mark: | :heavy_check_mark: |                           Boolean NMFk                                           |   [Link](examples/BNMFk/BNMFk.ipynb) |       :white_check_mark:       |
 
 ### TELF.pre_processing
 
 | **Method** | **Multiprocessing** |       **HPC**       |                           **Description**                          | **Example** | **Release Status** |
 |:----------:|:-------------------:|:-------------------:|:------------------------------------------------------------------:|:-----------:|:------------------:|
 |   Vulture  | :heavy_check_mark:  | :heavy_check_mark:  |         Advanced text processing tool for cleaning and NLP         |  [Link](examples/Vulture)  | :white_check_mark: |
 |   Beaver   | :heavy_check_mark:  | :heavy_check_mark:  |        Fast matrix and tensor building tool for text mining        |  [Link](examples/Beaver)  | :white_check_mark: |
-|  iPenguin  |                     |                     |         Online Semantic Scholar information retrieval tool         |             |       :soon:       |
-|    Orca    |                     |                     | Duplicate author detector for text mining and information retrival |             |       :soon:       |
-|            |                     |                     |                                                                    |             |                    |
 
-### TELF.post_processing
-
-| **Method** |                       **Description**                      | **Example** | **Release Status** |
-|:----------:|:----------------------------------------------------------:|:-----------:|:------------------:|
-|   Peacock  | Data visualization and generation of actionable statistics |             |       :soon:       |
-|    Wolf    |              Graph centrality and ranking tool             |             |       :soon:       |
-|    Fox    |              Report generation tool for text data            |             |       :soon:       |
-|    SeaLion    |              Generic report generation tool            |             |       :soon:       |
 
 ### TELF.applications
 
 | **Method** |                            **Description**                           | **Example** | **Release Status** |
 |:----------:|:--------------------------------------------------------------------:|:-----------:|:------------------:|
 |   Cheetah  |                        Fast search by keywords and phrases                       |    [Link](examples/Cheetah)         |       :white_check_mark:      |
-|    Bunny   | Dataset generation tool for documents and their citations/references |             |       :soon:       |
-|    Termite   | Knowladge graph building tool |             |       :soon:       |
-
 
 
 ## How to Cite T-ELF?
 If you use T-ELF please cite.
 
 **APA:**
 ```latex
-Eren, M., Solovyev, N., Barron, R., Bhattarai, M., Truong, D., Boureima, I., Skau, E., Rasmussen, K., & Alexandrov, B. (2023). Tensor Extraction of Latent Features (T-ELF) (Version 0.0.20) [Computer software]. https://doi.org/10.5281/zenodo.10257897
+Eren, M., Solovyev, N., Barron, R., Bhattarai, M., Truong, D., Boureima, I., Skau, E., Rasmussen, K., & Alexandrov, B. (2023). Tensor Extraction of Latent Features (T-ELF) [Computer software]. https://doi.org/10.5281/zenodo.10257897
 ```
 
 **BibTeX:**
@@ -226,4 +191,26 @@ ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 
 ## Developer Test Suite
-Developer test suites are located under [```tests/```](tests/) directory. Tests can be ran from this folder using ```python -m pytest *```.
+Developer test suites are located under [```tests/```](tests/) directory. Tests can be ran from this folder using ```python -m pytest *```.
+
+## LANL HPC Installation Notes
+
+### Chicoma
+```shell
+# replace <path to your conda environments under projects> with your own path below.
+conda create --prefix=<path to your conda environments under projects> python=3.11.10
+source activate <path to your conda environments under projects> # or use conda activate <...>
+pip install .
+python post_install.py --gpu --hpc-conda
+```
+
+### Darwin
+```shell
+salloc -n 1 -p shared-gpu
+module load openmpi
+module load miniconda3
+conda create --name TELF python=3.11.10
+conda activate TELF # or <source activate TELF>
+pip install .
+python post_install.py --gpu --hpc
+```

TELF/__init__.py

@@ -1,5 +1,5 @@
 from .version import __version__
 import sys
-sys.path += ["factorization/"]
-sys.path += ["pre_processing/"]
-sys.path += ["applications/"]
+sys.path += ["factorization"]
+sys.path += ["pre_processing"]
+sys.path += ["applications"]

TELF/applications/__init__.py

@@ -1,5 +1,5 @@
 import sys
-sys.path += ["Cheetah/"]
+sys.path += ["Cheetah"]
 
 # Cheetah
 from .Cheetah.cheetah import Cheetah