Readiness Discussion: SDK 1.0

[aaronsteers]

Question for discussion: Are we ready to release our 1.0?

Context

The SDK is now used in over 250 repos and is nearing 50 unique contributors.

To the @meltano/meltano-team and to our contributors:

1. Is there anything critical that we are missing feature-wise?
2. Any breaking changes or refactoring we'd want to do before dropping the v1.0 release tag?

---

As originally discussed in GitLab.

GitHub v1.0 Milestone

[aaronsteers]

Pulling from GitLab and more recent discussions:

• Make sure context arg is present in all applicable methods.
• Rename rows to records, esp. in function signatures: change: breaking rename changes: row->record (BREAKS) #966
• (Optional) Support LOG_BASED replication for DB-type sources: Support LOG_BASED replication method #304
• (Optional) Support the ACTIVATE_MESSAGE unofficial singer extension: Add support for ACTIVATE_VERSION message types #18, 'Hard and soft deletes' docs (suggested in "Add 'at least once' implementation info to SDK docs") #392

[edgarrmondragon]

Some things I would consider blockers for a stable release:

• Refactor "Singer interop" helpers to a public singer-sdk.singer module #208
• Consider refactoring external dependencies #928

Other "nice to have", maybe breaking, changes:

• feat: Emit metrics with a dedicated logger #933
• Not yet logged, but consider current --about --format=markdown vs simpler --about=markdown

Developer QOL:

• [Feature]: Consider not using a source that requires authentication for "external" tests in CI #877

[sbalnojan]

There might have already been a discussion about this, and I don't know the history, but how about "the name"?

What I mean: Inside the docs there is currently only one place which tells me that it's the "SDK for taps and targets", so maybe if we start to use something like tt-sdk it becomes clearer what I can do with it?

Or better the dsdk (SDK for moving data) because the stream maps are also developed using the SDK as far as I can tell.

After all, Meltano lives from being customized & extended, and we need to get people pointed into the right direction right off the bat.

[edgarrmondragon]

@sbalnojan we probably want to still communicate that it's used to create Singer-compatible taps and targets, so maybe meltano-singer-sdk? Or, if we plan to go well beyond Singer maybe meltano-flow-sdk?

[sbalnojan]

@edgarrmondragon I get that point, true, we need to communicate that. I would just try to decouple it. I would prefer something like "meltano-flow-sdk" (I like that), and put a huge sign right next to it "built on the open standard Singer".

Because this is the only thing that enables people to fully use Meltano. Every single "production ready" Meltano user will need to build 1-2 custom taps or targets, so there's no way around it. And everyone evaluating Meltano as a solution will ask "can I build custom taps/targets?", so I think clear communication is crucial here.

[kgpayne]

Things I would ❤️ to see before v1.0:

• Test framework & comprehensive standard tests (in progress)
• Tap/Target versioning, at least for MeltanoLabs plugins (in progress)
• SDK docs revamp, in line with @sbalnojans' excellent work on the Meltano docs
• Refactor connectors out of Stream/Sink classes (resolves use cases where Taps need to authenticate to perform discovery and also mocking connections for Tap/Target testing)

[tayloramurphy]

@aaronsteers we should probably include this issue for v1

• Feature: Returning informative exit codes #1409

[kgpayne]

@edgarrmondragon suggested in Office Hours:

• making SQLAlchemy optional (breaking change)

[edgarrmondragon]

Also cryptography, which we only use for the JWT authenticator.

[edgarrmondragon]

We should also make sure all breaking changes are documented in https://sdk.meltano.com/en/latest/deprecation.html

[kgpayne]

Another refactor that would be worth while would be to pull the BATCH file handling methods off of the Stream class onto a standalone BatchIO class (similar to the connections conversation). I ran into a case with target-snowflake where I wanted to reuse the batch methods available to a Stream in my Sink class 🤔