Design API

Goal of this task is to ensure we are compatible with the studio. Few pointers:

• data stream doesn't need a design API since it relies on dynamic data sampling to fulfill that "lack"

edit: checked out with [stream] team and schema handling can arrive but is not yet planned or needed.

• data preparation doesn't really need a design API since it can use the sampling as well

edit: [preparation] can probably migrate to use dynamic schema evaluation (TDP-4521)

• data catalog can need some design API but it will likely be more linked with the UI completion and data navigation than a dedicated API

edit: [datalog] can infer the schema from the sampling as well so no need of a dedicated API

• the studio requires a design schema definition (input/output)

edit: the [stream] logic sounds doable to start

High level idea: try to do the same as for data stream into the studio and avoid to require any schema definition for component developers.

Note: there is an optional @DiscoverSchema action in the framework which can allow to define explicitly a schema, it requires a better Schema API definition (don't use avro API but create our own API more centered around our usage)

Outcome of this task is a component family (source and processor) fully working into the studio without a lot of new API - note: only relevant if creating design API.