Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
None
-
All
-
Small
Description
Steps to reproduce or context:
When one service requests on component-server the list of all configuration types (see the example below) the response is compressed (gzip) and send by chunk.
Using the endpoint on the k8s service tpsvc-remote-engine or trough the internal ingress engine.$env.internal.lan does not give the same outcome all the time.
"Sometimes" the response one gets from the service is binary data instead of json. This has been observed on remote engine gen 2 installation in aws europe, but also (from tpd team) other environments.
Using the service on the ingress (traefik in between the caller and the k8s service) the data returned are always in text/json form.
Observed results:
Examples below come from the aws eu environment, the engine is a remote engine gen 2
export ENGINE_ID=7279f7cb-55ba-4b34-8fd6-db369c9be394
export TENANT_ID=e30a1384-69df-46db-ba0c-33afcd5126c3
kubectl run curl-engine-test -n ${KUBE_PS1_NAMESPACE:-app} -it \
--restart Never --rm \
--image=radial/busyboxplus:curl -- \
curl -i \
-H "X-Talend-remote-engine-id:$ENGINE_ID" \
-H "x-talend-tenant-id:$TENANT_ID" \
-H "x-talend-service-id:tcomp" \
-H "Accept: application/json" \
"http://tpsvc-remote-engine-server/api/v1/configurationtype/index?language=en&lightPayload=false"
Returns
X-Talend-Response-From-Engine: true
Content-Encoding: gzip
Talend-Vault-Cache: true
Talend-Vault-Cache-Time: 0
Date: Fri, 28 Feb 2020 13:14:46 GMT
vary: accept-encoding
Transfer-Encoding: chunked
Content-Type: application/json
�}{_۸��W�/w�{�9!���.�sg�4
�@��ʴ�qlC
:{�?�-�{�������_^4]����[ճ���\��+\�a�+�q��k�?��>�õ�w��<�\�\�1��,k���q�jw���_��Ƒ��p�_�
4*�ٷ�2Ne�loߌ
�@�d�#����q�;���g=�����XC�����y|�{�ȶ�ؓ'߀K�<sܷ��w�����x#[ T����vͺ7�mmo�6
<�,@C��v@�vh�ٚ*7��g��;
Doing the exact same call using the ingress does not give the same outcome. Here the output will be real json, traeifk is able to uncompress data on the fly.
- In both case we have the vary: accept-encoding http response header
- doing the call on the ingress (traefik) the Content-Encoding: gzip is set, data are in plaintext
- doing the call on the tpsvc-remote-engine-server (the k8s service) the Content-Encoding: gzip is set, data are compressed
Expected results:
If the request does not contain the "Accept-Encoding: gzip" header the response should not be compressed
Remarks / Analysis already done
The issue seems to be on the component-server that caches the response whatever the header "Accept-Encoding" is present or not. So if there is a call to the "/api/v1/configurationtype/index?language=en&lightPayload=false" result is put in the cache using the query parameters and the language http header (see https://github.com/Talend/component-runtime/blob/78bbe2710764343b617dbece90cc0e9f24106087/component-server-parent/component-server-vault-proxy/src/main/java/org/talend/sdk/component/runtime/server/vault/proxy/service/jcache/VaultProxyCacheKeyGenerator.java#L58)
Using "tacokit/component-server:1.1.15" locally (the version on which the issue has been found in aws eu and also on integration)
(in the docker-compose stack the service is named "tcomp")
curl -i "http://tcomp/api/v1/configurationtype/index?language=en&lightPayload=false" -m 10 -H "Accept: application/json"
gives
HTTP/1.1 200 connection: keep-alive Date: Wed, 17 Mar 2021 15:53:01 GMT Keep-Alive: timeout=60 vary: accept-encoding Talend-Vault-Cache-Time: 2 Talend-Vault-Cache: true Content-Type: application/json Transfer-Encoding: chunked {"nodes":{"cHVsc2FyI1B1bHNhciNkYXRhc3RvcmUjUHVsc2FyRGF0YVN0b3Jl":{"actions":[{"displayName":"Check connection","family":"Pulsar","name":"HEALTH_CHECK","properties":[{"displayName":"store","metadata":{"ui::gridlayout::Advanced::value":"timeout","ui::gridlayout::Main::value":"brokerURL|useTLS |authentication","documentation::value":"Apache pulsar connection.","action::healthcheck":"HEALTH_CHECK","configurationtype::name":"PulsarDataStore","configurationtype::type":"datastore","definition::parameter::index":"0"},"name":"store","path":"store","placeholder":"store","type":"OBJECT"} ,{"displayName":"Authentication","metadata":{"ui::gridlayout::Advanced::value":"authenticationType|basic","documentation::value":"Pulsar authentication."},"name":"authentication","path":"store.authentication","placeholder":"authentication","type":"OBJECT"},{"defaultValue":"NONE","displayNam e":"Authentication type","metadata"
Redo the same call with "Accept-Encoding: gzip"
curl -i "http://tcomp/api/v1/configurationtype/index?language=en&lightPayload=false" -m 10 -H "Accept: application/json" -H "Accept-Encoding: gzip"
HTTP/1.1 200 connection: keep-alive Date: Wed, 17 Mar 2021 15:53:01 GMT Keep-Alive: timeout=60 vary: accept-encoding Talend-Vault-Cache-Time: 1 Talend-Vault-Cache: true Content-Type: application/json Transfer-Encoding: chunked {"nodes":{"cHVsc2FyI1B1bHNhciNkYXRhc3RvcmUjUHVsc2FyRGF0YVN0b3Jl":{"actions":[{"displayName":"Check connection","family":"Pulsar","name":"HEALTH_CHECK","properties":[{"displayName":"store","metadata":{"ui::gridlayout::Advanced::value":"timeout","ui::gridlayout::Main::value":"brokerURL|useTLS |authentication","documentation::value":"Apache pulsar connection.","action::healthcheck":"HEALTH_CHECK","configurationtype::name":"PulsarDataStore","configurationtype::type":"datastore","definition::parameter::index":"0"},"name":"store","path":"store","placeholder":"store","type":"OBJECT"} ,{"displayName":"Authentication","
The issue is also visible with "component-server:1.1.27" version used on the ce4d and latest remote engine gen 2 release.
The issue should be moved to TCOMP project and the fix should consist in taking into account headers like "Accept" (if we can deliver other things than json) and "Accept-Encoding"