Details
-
Bug
-
Status: Done
-
Major
-
Resolution: Fixed
-
None
-
All
-
DQ20 CN 5
-
Small
Description
what is working
Storage | Spark Framework | Result |
---|---|---|
Azure Storage | local spark mode | ![]() |
Azure Storage | HDInsight | ![]() |
Amazon S3 bucket | local spark mode | ![]() |
what is not working
Storage | Spark Framework | Result | |
---|---|---|---|
Azure Storage | Azure Databricks | ![]() |
|
Amazon S3 bucket | AWS Databricks | ![]() |
|
Amazon S3 bucket | AWS EMR | ![]() |
will be fixed in TDQ-18366 |
HDFS | local spark mode | ![]() |
will be fixed in TDQ-18367 |
HDFS | HDP | ![]() |
will be fixed in TDQ-18367 |
HDFS | CDH | ![]() |
will be fixed in TDQ-18367 |
we want to support feature importance viz on the cluster for tMatchModel, the following problems need be fixed
issue4 don’t support to store on S3 bucket if running on Amazon EMR because the job can’t run well on the EMR cluster
issue5 test on HDFS+local spark mode/real cluster(HDP) after TDQ-18063 done
more information see sub-tasks
Acceptance Criteria
In ALL the following scenarios, download the pdf file and check that it can be read correctly and that the content is the one expected (same as in local mode).
- Scenario 1 Azure Storage
Given available Azure storage (need an Azure account first)
When run the tMatchModel job on Azure Databricks, "Model location" and "Model explanation" are enabled
Then the job run successfully and generate the feature importance viz PDF file in the assigned location, an image in the PDF file
- Scenario 2 Azure Storage
Given available Azure storage (need an Azure account first)
When run the tMatchModel job on Azure Databricks, "Model location" is enabled but "Model explanation" is NOT enabled
Then the job run successfully and NO the feature importance viz PDF file generated in the assigned location
- Scenario 3 Amazon S3
Given available S3 bucket (need an Amazon account first)
When run the tMatchModel job on AWS Databricks, "Model location" and "Model explanation" are enabled
Then the job run successfully and generate the feature importance viz PDF file in the assigned location, an image in the PDF file
- Scenario 4 Amazon S3
Given available S3 bucket (need an Amazon account first)
When run the tMatchModel job on AWS Databricks "Model location" is enabled but "Model explanation" is NOT enabled
Then the job run successfully and NO the feature importance viz PDF file generated in the assigned location
Attachments
Issue Links
1.
|
Azure Databricks |
|
closed | liu xinquan |
|
|||||||
2.
|
AWS Databricks |
|
closed | liu xinquan |
|
|||||||
3.
|
Amazon EMR |
|
Canceled | liu xinquan |
|
|||||||
4.
|
test on HDFS+local spark mode/real cluster(HDP/CDH) after TDQ-18063 done |
|
Canceled | liu xinquan | ||||||||
5.
|
backport TDQ-17479 to maintenance/7.3 and patch/7.3.1 |
|
closed | liu xinquan | ||||||||
6.
|
backport TDQ-18049 to maintenance/7.3 and patch/7.3.1 |
|
closed | liu xinquan | ||||||||
7.
|
backport TDQ-17784 to maintenance/7.3 and patch/7.3.1 |
|
closed | qiong li | ||||||||
8.
|
create temp cumulative 731 patch for QA (TDQ-17479/TDQ-18049/TDQ-17784) |
|
closed | liu xinquan | ||||||||
9.
|
[QA]Release on Monthly: TDQ-17479/TDQ-18049 |
|
closed | yunjie gao |