Go-Jek: Using Machine Learning for Dynamic Pricing and Forecasting

Using Google Cloud Platform and Google Maps Platform, GO-JEK has established itself as the leading on-demand multi-service platform and one of the few unicorn businesses in Southeast Asia.

“Google Maps Platform is currently the core platform that helps us find optimized routes and estimated arrival times for our drivers.”

— Ajey Gore, Group Chief Technology Officer, GO-JEK —

Traffic congestion is a fact of life for most Indonesians. National roads and related infrastructure are buckling to support 260 million people, of whom about 10 million reside in the capital, Jakarta. To minimize delays, Indonesians largely rely on motorbikes, including motorbike taxis, known as ojeks, for their daily commute to work or personal activities.

Founded in 2010 and headquartered in Jakarta, GO-JEK started as an ojek call center. The organization has fueled demand for the service to become one of the few “unicorns” — privately held startups and over $1 billion worth of holdings — in Southeast Asia.

Since 2010, GO-JEK has been collecting data to understand customer behavior and in 2015 launched a mobile app that includes ride-hailing, food and merchandise delivery. “Demand for the app is huge and we’re growing really fast,” said Willem Pienaar, Data Science Platform Team Leader at GO-JEK.

GO-JEK continues to listen to its customers and collect user feedback to feed into the development of new products. The business now has more than 18 products, a loyalty program, and a crypto service in its app, as well as other products outside of the app. Customers have downloaded the app nearly 108 million times.

GO-JEK today operates in 167 cities and districts across Indonesia and is the largest player in many key industries. “For ride-hailing alone, we have over 1 million drivers on our platform, and on a typical day we would have hundreds of thousands of drivers online at the same time,” says Pienaar. its customers. “We also have over 300,000 sellers on our food delivery platform. “We call them merchants because they are not just restaurants, they are parents who sell food from their garage. Our platform supports socioeconomic mobility in Indonesia by helping people lift themselves out of poverty. Furthermore, our payment service has become one of the leading e-money platforms in Southeast Asia.”

Key Ecosystem of Google

The business has heavily relied on the skills and expertise of its technology team, and on selecting the right technologies, to grow and to expand into new markets. GO-JEK helps to run its applications and data in Google Cloud due significantly to an ecosystem comprising of Google Cloud Platform, and other technologies.

The business relies heavily on the skills and expertise of the technology team as well as the selection of the right technology to grow and expand into new markets. GO-JEK ran its applications and data in Google Cloud, largely due to an ecosystem that included Google Cloud Platform and other technologies.

Ajey Gore, Chief Technology Officer at GO-JEK explains: “We had to find the right driver to meet the requirements – whether it was transporting people or food or other items from location A to location B. - as fast as possible". Google Maps Platform is now the core tool that helps us find optimized routes and estimated arrival times for our drivers. "

Google Cloud Platform solutions play an integral role in enabling GO-JEK to capture and analyze data related to 1 million drivers across 167 counties and cities as well as 2.5 million customers. “For example, we ping each of our drivers every 10 seconds, which means 6 million pings per minute and 8 billion pings per day,” Gore said. “If you look at the scale and number of customer interactions, we generate about 4TB to 5TB of data per day. We need to leverage this data to let our drivers know where customer demand is strongest and how to get there.”

Cloud Professional Services, we have a deeper understanding of Google technologies, better collaboration, smarter design, and faster time to market.”

—Willem Pienaar, Data Science Platform Lead, GO-JEK—

Expanding group in the international market

As GO-JEK expands into international markets, data science and machine learning becoming increasingly important for businesses. “Data Science plays an important role in understanding our customers,” Pienaar explains. “With our international expansion in 2018, we wanted to understand the unique needs of our customers in different markets. Furthermore, when there are so many decisions to be made in real time, machine learning is crucial.”

Pienaar investigated how GO-JEK data scientists spend their time and whether they have access to the tools they need. “One of our data scientists asked if we could help him optimize his time,” he explains. “This engineer represents many of the engineers in our organization and is working on projects in areas such as fraud detection, personalization, driver allocation, and forecasting. However, he spent more time on engineering than exploring data and creating features and models.”

By looking at data scientists' projects across three dimensions: data sourcing, feature engineering, and machine learning, Pienaar's team identified the need to establish a centralized data platform.

GO-JEK then worked closely with Google Cloud Professional Services (https://cloud.google.com/consulting) to further explore and refine your ideas. “The Google Cloud Professional Services team has been instrumental in driving the success of several key projects for the GO-JEK data science team,” says Pienaar. “The team brings a wealth of experience in managing high-impact projects from inception through to production. Ultimately, thanks to Google Cloud Professional Services, we have a deeper understanding of Google's technologies, better collaboration, smarter design, and faster time to market.”

The Professional Services team has organized seminars, architecture and design conferences, as well as in-depth research into real-world technology. The team members then worked with GO-JEK to develop a POC using technologies such as Apache Beam running on Cloud Dataflow, Cloud Bigtable, and Cloud Bigtable. BigQuery, which forms the basis of the GO-JEK feature engineering and storage platform.

Other technologies used in the platform – which will collect and store data from GO-JEK products and other groups – include Apache Kafka, Cloud Pub/Sub and Cloud Storage. Those teams will benefit from automated reporting, storage, monitoring, event alerting and security through the platform, while data science and machine learning teams can access their data. needed to create insights.

With feature generation, standardization, and consistency addressed through the combination of Cloud Dataflow and Apache Beam, enterprises had to choose a feature hosting environment. “We chose BigQuery to store our training data because of its high scalability and the fact that it is a fully cloud-based service which means you don’t have to manage any facilities,” says Pienaar. What infrastructure? “In addition, it is tightly integrated with other Google Cloud services.”

“We believe in the power of unity, and over the past two years we've worked closely with Google to unlock new limits and possibilities. Who would have thought that we would now become the leading application for transportation, food and other services in Indonesia? ”

—Ajey Gore, Group Chief Technology Officer, GO-JEK—

To serve, GO-JEK introduced two datastores: Cloud Bigtable and Cloud Memorystore for Redis. Pienaar said: “Cloud Bigtable is really a game changer for us. “It allows us to seamlessly access feature data at less than 10 millisecond latencies and handle very high loads, so you can write and read that data up to 10,000 times per second combined per node. and if you want to scale, you just need to add more nodes.

“Meanwhile, Cloud Memorystore for Redis allows us to respond to scenarios like updating features 200,000 or 300,000 times per second,” he added.

Feature-serving API allows quick feature lookup from Cloud Bigtable or Cloud Memorystore and provides caching.

With all this data logged and extremely useful to data scientists, GO-JEK dumps the metadata into a PostgreSQL database. From there, businesses use Google Data Studio to visualize how well specific features do in predicting results or optimizing for goals. GO-JEK has also created a feature explorer that data scientists can use to find features, select them in BigQuery, and train their models.

Cloud Machine Learning supports dynamic pricing

With the data source and feature engineering building blocks established, GO-JEK has empowered its data scientists to apply scalable machine learning models to deliver dynamic pricing and innovation. other. Pienaar explains: “We introduced the Cloud Machine Learning Engine (https://cloud.google.com/ml-engine), which is tightly integrated with BigQuery as our feature and raw data warehouse. Cloud Machine Learning Engine is a scalable, managed service that allows GO-JEK data scientists to train and serve TensorFlow models.

The GO-JEK architecture also incorporates Cloud Dataflow to infer future flows and needs based on Apache Kafka. Businesses can now effectively predict changes in demand due to events like weather changes or price changes to adjust how drivers react. It can also personalize the application home screen while responding to service-level requests with 30 millisecond latency and up to 10,000 requests per second throughput.

GO-JEK engineers are now only interested in using Cloud Datalab to build machine learning models; find features in the feature explorer; use BigQuery to create a training group; and use Cloud Machine Learning Engine to train their models.

“The impact for GO-JEK is that our data scientists can deliver projects faster, our customer experience has improved a lot, we need less data scientists per customer as we expand into new markets and we spend less time and money on infrastructure.” Pienaar.

Gore adds: “We believe in the power of unity, and over the past two years, we've worked closely with Google to unlock new limits and possibilities. “Who would have thought that we would now become the leading application for transportation, food and other services in Indonesia?”