Visualized Data Development Platform
Demand pain points
Data access
• Offline, quasi-real-time and can not covered by one platform;
• With so many accessed source system database and interface types, some can not support and need to be developed independently;
• Low access efficiency; large number of tables; no templates, batch collection, non-distributed;
Process control
• Lack of data model design and management platform;
• Market data model design tools docked to open source Hadoop with good commonly used databases;
• Data model design cannot automatically convert ETL tasks with few types of data conversion rules;
• Data consolidation requires code development, have technical threshold
Task Scheduling
• Lack of visualized task orchestration and scheduling;
• Less task types
• Limited number of concurrent tasks;
• The excessive number of tasks makes the process design diagram easy to be confused and the relationship is difficult to clarify
SQL development
• Lack of automated SQL review process and irregular SQL affect system operation;
• Inability to visualize the analysis of SQL lineage makes traceability difficult;
• lack SQL version management leads to difficulties in file management;
• Not supporting SQL IntelliSense affects development efficiency;
TDS Visualized Data Development Platform
◆ Product capability: Provide diversified data-related tools to support the whole process of visualized data development, monitoring, operation and maintenance, and alarming
◆ Product goal: To realize data unification and visualization, build enterprise-level data lake, data warehouse and data mart
◆ Core functions: data access, data integration, data services
◆ Technical design: ELT (Extract - Load - Transform)
◆ Extract and load based on graphical steps for the improvement of efficiency
◆ SQL-based Transform is more flexible and efficient in using big data platforms
Product name
Waterdrop
TDH data platform development desktop IDE
SQL IDE for ArgoDB and KunDB
SQLBook
SQL development and execution
SQL lineage static analysis
use DevOps to management SQL development
Workflow
Visualization design, task and taskflow configuration
Distributed data task scheduling
Collaborative development and graphical operations and maintenance
Transporter
Supported integrated data sources and targets
Real-time Synchronization,Data Transform
Collaborative development and graphical operations and maintenance