Log time elapsed for each ingest step

[naglepuff]

@kheal @eecavanna

At the infrastructure sync meeting on 8/21/2025, we discussed a recent ingest taking 50% longer to run than previous ingests in the same environment. One potential cause could be new records in mongo. While this may not be the cause of this particular spike in ingest time, it is worth investigating.

In order to make investigating this easier, we could add additional logging to the ingest process to report the time elapsed for each ingest step.

[kheal]

@naglepuff and @eecavanna

I looked more closely at the ingest logs and compared them to when I submitted the data to dev. According to my notes, I loaded them on 8/15 at 6pm PST. According to the slack bot reports of ingest time, the ingest to dev for 8/15-8/20 looked normal (120-140 minutes). So I doubt the new records I mentioned are the issue after all.

[eecavanna]

FYI, for performance profiling in nmdc-runtime, I recently started using Pyinstrument.

Although, during today's infrastructure sync, I demo-ed it using the HTTP request/response cycle, it can also be used for "regular" Python scripts, as shown here: https://pyinstrument.readthedocs.io/en/latest/guide.html