Industrial Data / SaaSTechAgent (Suwen Intelligence)2024-09-27

MatrixOne Builds a Hyper-Converged AIGC Data Foundation Platform

TechAgent replaced MySQL, MongoDB, ElasticSearch, Faiss, and ClickHouse with a single MatrixOne instance, cutting customer delivery time from 2 months to 1 week.

TechAgent (Suwen Intelligence)

Shenzhen Suwen Intelligence, operating as TechAgent, was founded in 2017 and provides industrial-chain intelligence data services to dozens of leading manufacturers and government bodies.

2mo→1wk
Delivery time
hrs→min
Processing speed
5→1
DB components
K8s
Cloud-native

The challenge

TechAgent is a data-processing business at its core, delivered as research reports, data SaaS and APIs. All three rest on one in-house data platform, and to serve that range of delivery forms the platform had grown complex enough to be a burden in both operations and iteration.

Upstream sits a large collection of acquisition programs — web crawlers for public data, APIs for commercial data, preprocessing that extracts data from files — all structured and landed in MySQL. Some of it must pass through an LLM before use, and because of how those algorithms behave the output is deeply nested JSON that MySQL's JSON support could not handle, so a MongoDB instance was introduced solely to store and query it.

Search is a core capability, and it needs both keyword full-text and semantic vector retrieval. So ElasticSearch and Faiss were added, one for each.

As the business grew and sources multiplied, MySQL itself became the constraint: write concurrency had to be absorbed by caching in front-end code, and single tables past ten million rows forced manual sharding to keep queries acceptable. For report generation and heavy analysis, a ClickHouse instance was added on top of that.

The solution

MatrixOne is a cloud-native converged database built on Kubernetes and shared storage, supporting OLTP, OLAP, time-series, vector and search workloads together while staying highly MySQL-compatible. For a workload built deeply on large models and document data, it can serve as a single data foundation and take most of the architectural complexity out of delivery and operations.

As an HTAP engine it covers what MySQL and ClickHouse were doing separately, which also removes the scheduled ETL that moved data between them — analysis is written as SQL over views instead. MySQL compatibility meant TechAgent's Django and SQLAlchemy stack connected without change, so migrating off MySQL and ClickHouse was the natural first step.

MatrixOne carries native JSON and vector types. Columns of vecf32/vecf64 or JSON can be created in any table, with vector index and search functions and JSON parsing functions provided, which covers most of what MongoDB and Faiss were there for.

It also supports inverted indexes — the core of ElasticSearch full-text search — and inverted and vector retrieval can be combined in one query with a user-tunable weighting. The reason the original architecture needed both ElasticSearch and Faiss was to fuse keyword and semantic search at the application layer; doing it in the database removes that layer entirely.

The outcome

Before MatrixOne, delivering to a large customer took an average of two months, and the hardest part was deploying and tuning the database layer. After the migration the stack is as simple to operate as a single MySQL instance, and with help from the MatrixOrigin cloud-native team the application and database became a single integrated delivery — bringing that cycle down to one week.

End-to-end data processing moved from hours to minutes.

TechAgent CEO Wang Wei: the architecture had grown by patching whatever was missing rather than by design, and that caught up with iteration and delivery; consolidating it cleared the accumulated technical debt in one step.

Solution Architecture

Data sources
  • Industrial-chain company and product data
  • Multi-source public information
  • Customer delivery datasets
  • Historical labelling and QA records
MatrixOne Intelligence
  • Converged engine replacing multiple databases
  • Unified multi-modal storage
  • Vector and full-text retrieval
  • Processing and labelling pipeline
  • Elastic scaling on demand
Business applications
  • Industrial-chain intelligence services
  • Shorter customer delivery cycle
  • Higher data processing throughput
  • One platform replacing a complex stack

Related solutions and products