HSC hiện đại hóa cơ sở hạ tầng công nghệ để cải tiến việc cung…
Traveloka: Switch to Google Cloud Platform for Powerful Big Data Analytics
SUMMARY
Business problem:
- Debugging problems in Kafka clusters proved difficult and time consuming
- Adding more nodes to MongoDB required a lengthy rebalancing process – and the pool quickly ran out of disk space
- Enterprises can only store data for 14 days in MemSQL due to memory limitations, while queries sometimes return out of memory errors.
Solution:
- Google Cloud Platform
- BigQuery
- Cloud Pub/Sub
- Cloud Dataflow
- Kubernetes Engine
- Cloud Storage
- Cloud Composer
- Cloud SQL
Result:
- Engineers relax, take their time to bring value to the business
- Record more than 99.9% availability
- Warehouse 400TB (about 500 billion lines) of data

With Google Cloud technologies such as BigQuery, Traveloka has established a data architecture that meets all performance and availability requirements and enables businesses to gain meaningful insights and can act from large volumes of data. Collect and analyze data in real time for enterprise-wide decision making.
Founded in 2012, Traveloka is a unicorn business that provides reservations for travel, dining and other options. The organization has grown to establish a presence in six ASEAN countries and employs more than 2,000 people, including 400 engineers. Traveloka aims to be a one-stop travel and lifestyle platform for Indonesians and is diversifying and personalizing its services. The business introduced features like car rental bookings and travel destination guides in 2018, and added a host of extras to existing services, such as status notifications flights for flight booking service. Notable last year was the launch of an online credit service.
Traveloka relies on data analytics to provide tailored, personalized services to consumers. This poses a major challenge to the enterprise data analytics team. This team must support the growing business need for actionable insights by collecting data from multiple sources, choosing the right framework for data analysis, managing multiple use cases, and delivering data. Real-time data for stream analysis and reporting. At the same time, businesses must scale infrastructure while reducing costs.
Analytics team activities should support business goals to increase agility and faster time to market for new features and apps. From a technology standpoint, this means speeding up development and delivery without compromising security.
“As part of the Google Cloud Platform data transfer analytics solution, support from BigQuery for streaming data is a key advantage for us in supporting the real-time analytics use case.”
—Rendy Bambang, Data Engineering Lead, Traveloka—
Data analytics not keeping pace
However, as the business expands, Traveloka's current data analytics environment cannot keep up. This has impacted online data processing that supports a number of use cases – including fraud detection, personalization, ad optimization, side-selling, A/B testing and calculation. ad-eligible – allows business analysts to track performance.
To run the data analytics pipeline, Traveloka relied on an architecture that includes Apache Kafka for ingesting user events, fragmented MongoDB to provide an operational datastore that spans multiple machines, and fragmented MemSQL for Real-time analytics queries. Traveloka processed data from Kafka through a Java user and stored it with the user ID as the primary key in MongoDB. For analysis, Traveloka used event data from Kafka and stored it in MemSQL, which is accessible to business intelligence tools.
“Cloud Pub/Sub is particularly convenient for us because – unlike the previous architecture, which required capacity planning for the entry of events – we can rely on its automation to handle volume and throughput changes without having to do anything.”
—Rendy Bambang, Data Engineering Lead, Traveloka—
Low latency and fully managed infrastructure
The business decides to explore the market and establish an alternative service needed to provide:
- Low end-to-end data latency in guaranteed service level agreement
- Fully managed infrastructure to relax engineers, help solve business problems (and spend less time on maintenance and firefighting), including resiliency or availability use of the 99.9% end-to-end system and automatically scale up storage and computing
These requirements are filtered into a necessity for a fully managed technology with low end-to-end latency, high performance and availability, and minimal operational demands.
Google Cloud Platform as a platform
Traveloka đã tiến hành đánh giá và kết luận rằng Google Cloud Platform đã cung cấp các dịch vụ và hiệu suất để hoạt động như nền tảng của kiến trúc dữ liệu.
Đối với dự án pipeline dữ liệu của mình, Traveloka đã triển khai môi trường cross-cloud kết hợp Cloud Pub/Sub (https://cloud.google.com/pubsub) quản lý nhắn tin thời gian thực để nhập dữ liệu sự kiện, Cloud Dataflow (https://cloud.google.com/dataflow) để xử lý dữ liệu được truyền trực tuyến và kho dữ liệu phân tích BigQuery để lưu trữ dữ liệu lịch sử và thực tế được tạo ra bởi các hoạt động của khách hàng, cũng như dữ liệu đã xử lý. Mỗi dịch vụ Google Cloud Platform đã giúp khắc phục các vấn đề cản trở pipeline trước đây.
BigQuery Analytical Data Warehouse is key to the new architecture. Rendy Bambang, Data Engineering Lead, Traveloka said: “As part of the Google Cloud Platform data transfer analytics solution, support from BigQuery for streaming data is a key advantage for us in supporting real-time analytics use case”. “Furthermore, we no longer have to worry about storing historical data for only 14 days because BigQuery stores all that data for us, with computing resources that automatically scale as required. we need."
“Cloud Dataflow's ability to create new pipelines and auto-scale without user intervention is a big plus for us, especially when we have to fill up a pipeline for processing. historical data management.”
—Rendy Bambang, Data Engineering Lead, Traveloka—
“Cloud Pub/Sub is particularly convenient for us because – unlike our previous architecture, which required capacity planning for event ingestion – we can rely on automated its partitioning to handle volume and throughput changes without any work,” added Bambang. “Ultimately, Cloud Dataflow's ability to create new pipelines and automatically scale without user intervention was a big plus for us, especially when we had to fill a pipeline to historical data processing.”
Cloud Dataflow's Apache Beam-based unified programming model eases the transition between batch and stream data processing, while its windowing and triggering functions allow for easy processing of slow incoming data.
400TB of data successfully stored
Cơ sở hạ tầng Google Cloud Platform hiện đang quản lý khối lượng lớn một cách nhanh chóng và tốt trong tổ chức với trên 99,9% tính khả dụng từ đầu đến cuối. Hơn 4TB dữ liệu mỗi ngày được chuyển vào Cloud Pub/Sub, trong khi BigQuery lưu trữ khoảng 400TB (khoảng 500 tỷ dòng) dữ liệu. Khoảng 250TB dữ liệu nằm trong Cloud Storage, trong khi 60.000 nhóm công việc được thực hiện mỗi ngày. Cloud Dataflow xử lý khoảng 2.500 công việc mỗi ngày, trong khi khoảng 1.500 biểu đồ sử dụng BigQuery được tạo bằng các công cụ kinh doanh thông minh.
BigQuery Warehouse is also integral to changes in how Traveloka gives its product teams access to data. Imre Nagi, Software Engineer, Data Team, Traveloka said: “In the past, when a product team requested data from our data warehouse, we simply gave them direct read access to the groups. or the board they need”.
However, this approach requires the customer system to be tightly coupled to the data storage technology and format, which means that any change to the technology or format requires updating. system. Furthermore, because access is at the group level, the data team cannot be sure that product groups are not accessing columns they are not authorized to. Finally, the data team finds it difficult to track and check what users are doing with the data.
Normalized data distribution
“Based on issues across the enterprise, we decided to build a standardized way to serve our data, which would later become our data provisioning API,” said Nagi.
API hiện phân phối hàng triệu bản ghi có tổng dung lượng vài gigabyte từ kho BigQuery đến hệ thống sản xuất theo yêu cầu. Cloud Composer (https://cloud.google.com/composer)lên lịch cho các truy vấn BigQuery để chuyển đổi dữ liệu thô thành các phiên bản tóm tắt và biên tập lại để chuyển vào các bảng trung gian và bảng cuối cùng đã được xử lý.
Cloud Storage cung cấp bộ nhớ tạm thời cho các kết quả query và xử lý quá trình gửi kết quả đến clients và Cloud SQL theo dõi các liên kết, state và metadata khác, trong khi API được lưu trữ trong các cụm Kubernetes do Google Kubernetes Engine quản lý. Các cụm Kubernetes (https://cloud.google.com/kubernetes) giao tiếp với Cloud Storage và Cloud SQL để lưu trữ kết quả và job metadata của các query do người yêu cầu thực hiện.
Issues solved
“With Google Cloud Platform technology, our new data provisioning API has successfully solved a number of issues during data delivery,” said Nagi. “We now have a clear contract API that standardizes how product teams access our data warehouse.”
Using the API means that product teams no longer access the physical layer of Traveloka's infrastructure, improving the data team's ability to audit data usage. Teams can also define column-level access controls, ensuring product teams use only the columns they need. Additionally, the API provides a standard yet flexible definition that other groups can use to query data. “We can now restrict how product teams access our data, while still allowing a wide variety of queries,” says Nagi. “Overall, we now have the flexibility, along with the security and control we need.”