[FLINK-38242] Improve upgrading documentation for SQL/Table API state compatibility and add Flink 2.x to compatibility table

[weiqingy]

What is the purpose of the change

This pull request improves the upgrading documentation by adding clear warnings about SQL/Table API state compatibility issues and including missing Flink 2.x versions in the compatibility table.

Brief change log

• Added warning that SQL/Table API jobs may have incompatible savepoints even during minor version upgrades due to query optimizer changes
• Added Flink 2.0.x and 2.1.x to compatibility table with note that state compatibility is not guaranteed between 1.x and 2.x

Verifying this change

After the update, the documentation will be as follows:

Does this pull request potentially affect one of the following parts:

• Dependencies: no
• The public API: no
• The serializers: no
• The runtime per-record code paths: no
• Deployment or recovery components: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 2130dad Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[davidradl]

@weiqingy this is a nice addition. I wondered can we say what state is not compatible between v1 and v2. I suspect it should work apart from serializing state that involves:

• some / all custom serializers
• Scala Case Class datatype
• POJOs using Kryo
• General Class Types using Kryo

Anything else we should add to the list? If this is the case it would really useful to document these restrictions to help Flink users know if there are likely to hit migration issues or not.