Data Loader for Domain Specific Datasets

[michelecatani]

Hello @hosseinfani @DelaramRajaei

It appears that the CLEF-IP track is not accessible through ir_datasets. It's possible to still build a loader but it would need to be markedly different from the other loaders in the repo. Would you like me to explore this option?

Alternatively, I could start building loaders for other datasets which are present in ir_datasets if we want to stay within that. The option would be to start building one for nfCorpus (medical) while I look for a legal dataset within ir_datasets.

[DelaramRajaei]

Hi @michelecatani ,

Not all the loaders need to be the same, and they don't have to come exclusively from the ir-dataset. It's easier when the information is in the ir-datasets, but it's fine if it's not.

I'd really appreciate it if you could add other loaders too. I've noticed in some papers on backtranslation that it works especially well in specific domains like law, legal, and medical. If you could create data loaders for datasets in these domains, that would be fantastic.

[michelecatani]

Hi all, I have completed the loader of nfCorpus and it's outputting the indexed file to whatever folder you pass to it. I have tested it with the beir/nfcorpus/test dataset. This dataset was very small, and I believe it was akin to what you were talking about when you mentioned a toy @hosseinfani . I am aiming to build two more loaders for the next meeting and I'll look to merge them all before then as well.

Before I merge the code, I would like to run it by either @hosseinfani or @yogeswarl as @DelaramRajaei mentioned that the previous loaders were built by the two of you. I had to make some changes to the code locally to get it to run and ideally I'd like to revert those changes before I merge so that it matches the other loaders. Are either/both of you available to meet virtually Tuesday or Thursday afternoon sometime after 1PM?

[yogeswarl]

Hi @michelecatani, Could you please send a Merge Request to Repair or point me to your version of code/ Repo, So I can take a look

Also provided me with a detailed guide on how and where can I fetch the dataset and run it for the test.

The easiest thing to do is Push your code to your version(fork) of Repair and send me the GitHub. I will run the test and ask you to send a Merge Request to Repair.

If you are stuck anywhere please feel free to reach out to me or Dr. @hosseinfani

Thanks

[michelecatani]

Sounds good @yogeswarl , I've put through the pull request, you can see it here: #43

You can also pull my fork from here: https://github.com/michelecatani/RePair

You don't need to get the dataset anywhere, it will be loaded from ir-datasets. However, you will need to instantiate an nfCorpus object. This is how I did it in Colab:

[yogeswarl]

Hello @michelecatani ,
I have tested out the code.

There are few changes we need to make to successfully integrate the nfCorpus to Repair as the main pipeline

Since I was unable to push changes to your repo. I will paste the code here.
Please add them to the respective files

param.py

# just after aol
  'nfCorpus' : {
        "index":'../data/raw/nfCorpus/lucene-index/',
        "index_item":['title'],
        'pairing': [None, 'docs', 'query'],
    }

main.py
Line 26:

if domain == 'nfCorpus': 
            from dal.nfCorpus import nfCorpus
            ds = nfCorpus(param.settings[domain], datapath, param.settings['ncore'])

nfcorpus.py
Line 8

from ds import Dataset 
# change to 
from dal.ds import Dataset

Let me know if you encounter any error. I would be happy to help.

[michelecatani]

Hi @yogeswarl I have implemented your suggested changes, thank you for those!

The updates are in this PR: #44

You'll notice there's a few commits. Some of the output files somehow got deleted somewhere so I had to reintroduce them. However, the final PR only contains changes to main.py, param.py, and the addition of nfCorpus.py.

Before the next meeting on Friday I will have two more loaders ready. I will provide them to you ready to test and upon your confirmation of success I can merge those too!

[yogeswarl]

Did you get a chance to run the nfCorpus in the main pipeline?

[michelecatani]

Hi @yogeswarl no, I have not had a chance to do so yet. I will do so tomorrow and let you know how that goes

[michelecatani]

Hi @yogeswarl , I tested it and it worked with the main pipeline. This is the output:

[michelecatani]

@yogeswarl Actually, there seems to be an error below but I don't think this applies to my changes. It references a stats.py file and the msmarco.passage, which I have not touched... if I have misinterpreted, let me know

[michelecatani]

@yogeswarl I have pushed a second loader for a trec dataset with Covid data. I have tested it with the main pipeline and it works and builds the index, however I get the same error I've posted above.

The new loader is in this fork too: https://github.com/michelecatani/RePair

[yogeswarl]

Great Job @michelecatani . Let me have a look at the code. Don't worry about the code. It is irrelevant error. We just need to comment out a line. I will give this as a task later on for you to rewrite.

Unfortunately I do not have write access to Repair anymore. Dr. @hosseinfani could you please merge this pull request I have tested this code.

[yogeswarl]

Hello @michelecatani, There is one thing I would like you to do before Dr. @hosseinfani can merge this code.

If you see in these screenshots:
trec-covid

nfcourpus

Both of these loaders have the same functionality as the aol-ia loader.
This is because they all come from ir-Datasets. Can you write a common loader where you pass only the corpus name. This way we can avoid creating the same file for different loaders.

Do let me know if you are stuck somewhere.

Here are the test cases: aol-ia, nfCorpus, tree-covid.
On passing these dataset, Create json files and build an index in their respective folder under raw/loader-name