Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-1880

Engine Server returns binary data instead of json (aka does not respect the compressed header)

Details

    • All
    • Small

    Description

      Steps to reproduce or context:

      When one service requests on component-server the list of all configuration types (see the example below) the response is compressed (gzip) and send by chunk.
      Using the endpoint on the k8s service tpsvc-remote-engine or trough the internal ingress engine.$env.internal.lan does not give the same outcome all the time.

      "Sometimes" the response one gets from the service is binary data instead of json. This has been observed on remote engine gen 2 installation in aws europe, but also (from tpd team) other environments.
      Using the service on the ingress (traefik in between the caller and the k8s service) the data returned are always in text/json form.

      Observed results:

      Examples below come from the aws eu environment, the engine is a remote engine gen 2

      export ENGINE_ID=7279f7cb-55ba-4b34-8fd6-db369c9be394
      export TENANT_ID=e30a1384-69df-46db-ba0c-33afcd5126c3
      kubectl run curl-engine-test -n ${KUBE_PS1_NAMESPACE:-app} -it \
      --restart Never --rm \
      --image=radial/busyboxplus:curl -- \
      curl -i \
      -H "X-Talend-remote-engine-id:$ENGINE_ID" \
      -H "x-talend-tenant-id:$TENANT_ID" \
      -H "x-talend-service-id:tcomp" \
      -H "Accept: application/json" \
      "http://tpsvc-remote-engine-server/api/v1/configurationtype/index?language=en&lightPayload=false"
      

      Returns

      X-Talend-Response-From-Engine: true
      Content-Encoding: gzip
      Talend-Vault-Cache: true
      Talend-Vault-Cache-Time: 0
      Date: Fri, 28 Feb 2020 13:14:46 GMT
      vary: accept-encoding
      Transfer-Encoding: chunked
      Content-Type: application/json
      
      �}{_۸��W�/w�{�9!���.�sg�4
      �@��ʴ�qlC
      :{�?�-�{�������_^4]����[ճ���\��+\�a�+�q��k�?��>�õ�w��<�\�\�1��,k���q�jw���_��Ƒ��p�_�
                                                                                          4*�ٷ�2Ne�loߌ
      �@�d�#����q�;���g=�����XC�����y|�{�ȶ�ؓ'߀K�<sܷ��w�����x#[                                          T����vͺ7�mmo�6
                                                            <�,@C��v@�vh�ٚ*7��g��;
      

      Doing the exact same call using the ingress does not give the same outcome. Here the output will be real json, traeifk is able to uncompress data on the fly.

      • In both case we have the vary: accept-encoding http response header
      • doing the call on the ingress (traefik) the Content-Encoding: gzip is set, data are in plaintext
      • doing the call on the tpsvc-remote-engine-server (the k8s service) the Content-Encoding: gzip is set, data are compressed

      Expected results:

      If the request does not contain the "Accept-Encoding: gzip" header the response should not be compressed

      Remarks / Analysis already done

      The issue seems to be on the component-server that caches the response whatever the header "Accept-Encoding" is present or not. So if there is a call to the "/api/v1/configurationtype/index?language=en&lightPayload=false" result is put in the cache using the query parameters and the language http header (see https://github.com/Talend/component-runtime/blob/78bbe2710764343b617dbece90cc0e9f24106087/component-server-parent/component-server-vault-proxy/src/main/java/org/talend/sdk/component/runtime/server/vault/proxy/service/jcache/VaultProxyCacheKeyGenerator.java#L58)

      Using "tacokit/component-server:1.1.15" locally (the version on which the issue has been found in aws eu and also on integration)
      (in the docker-compose stack the service is named "tcomp")

      curl -i "http://tcomp/api/v1/configurationtype/index?language=en&lightPayload=false" -m 10 -H "Accept: application/json"
      

      gives

      HTTP/1.1 200
      connection: keep-alive
      Date: Wed, 17 Mar 2021 15:53:01 GMT
      Keep-Alive: timeout=60
      vary: accept-encoding
      Talend-Vault-Cache-Time: 2
      Talend-Vault-Cache: true
      Content-Type: application/json
      Transfer-Encoding: chunked
      
      {"nodes":{"cHVsc2FyI1B1bHNhciNkYXRhc3RvcmUjUHVsc2FyRGF0YVN0b3Jl":{"actions":[{"displayName":"Check connection","family":"Pulsar","name":"HEALTH_CHECK","properties":[{"displayName":"store","metadata":{"ui::gridlayout::Advanced::value":"timeout","ui::gridlayout::Main::value":"brokerURL|useTLS
      |authentication","documentation::value":"Apache pulsar connection.","action::healthcheck":"HEALTH_CHECK","configurationtype::name":"PulsarDataStore","configurationtype::type":"datastore","definition::parameter::index":"0"},"name":"store","path":"store","placeholder":"store","type":"OBJECT"}
      ,{"displayName":"Authentication","metadata":{"ui::gridlayout::Advanced::value":"authenticationType|basic","documentation::value":"Pulsar authentication."},"name":"authentication","path":"store.authentication","placeholder":"authentication","type":"OBJECT"},{"defaultValue":"NONE","displayNam
      e":"Authentication type","metadata"
      

      Redo the same call with "Accept-Encoding: gzip"

      curl -i "http://tcomp/api/v1/configurationtype/index?language=en&lightPayload=false" -m 10 -H "Accept: application/json" -H "Accept-Encoding: gzip"
      
      HTTP/1.1 200
      connection: keep-alive
      Date: Wed, 17 Mar 2021 15:53:01 GMT
      Keep-Alive: timeout=60
      vary: accept-encoding
      Talend-Vault-Cache-Time: 1
      Talend-Vault-Cache: true
      Content-Type: application/json
      Transfer-Encoding: chunked
      
      {"nodes":{"cHVsc2FyI1B1bHNhciNkYXRhc3RvcmUjUHVsc2FyRGF0YVN0b3Jl":{"actions":[{"displayName":"Check connection","family":"Pulsar","name":"HEALTH_CHECK","properties":[{"displayName":"store","metadata":{"ui::gridlayout::Advanced::value":"timeout","ui::gridlayout::Main::value":"brokerURL|useTLS
      |authentication","documentation::value":"Apache pulsar connection.","action::healthcheck":"HEALTH_CHECK","configurationtype::name":"PulsarDataStore","configurationtype::type":"datastore","definition::parameter::index":"0"},"name":"store","path":"store","placeholder":"store","type":"OBJECT"}
      ,{"displayName":"Authentication","
      

      The issue is also visible with "component-server:1.1.27" version used on the ce4d and latest remote engine gen 2 release.

      The issue should be moved to TCOMP project and the fix should consist in taking into account headers like "Accept" (if we can deliver other things than json) and "Accept-Encoding"

      Attachments

        Activity

          People

            emmanuel_g emmanuel gallois
            mtallet Michael Tallet
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: