Localizing a marketplace with complex language requirements

Since 2015, Technokratos has been helping large businesses grow and develop through mobile apps, web services, and consulting. Like us, they develop digital services for a variety of industries, including fintech, ecommerce, etc.

Technokratos invited us to help develop a sub-project for their client, a large marketplace that can be called an alternative to online shopping services like Ozon and Aliexpress. They provide sellers with an opportunity to sell goods through their platform and organize delivery. 

When they approached Evrone, they were looking to localize their service for Uzbekistan. They had separated this activity into a separate sub-project, which they decided to build using what they already had. However, they needed to adapt their service to meet the needs of people living in Uzbekistan, taking into account their language preferences, pricing, currency, suppliers, and so on.

The task

Evrone provided Go developers for the project, who worked together with a strong and well-coordinated team of the project. Our task was to work on the search backend. High-functioning eCommerce search engines are critical for marketplaces — this includes not only the search for a product itself, but also search suggestions, identifying suitable categories, ranking, and collecting analytics for search queries. Our main tasks were to:

• Deploy a completely separate and independent infrastructure
• Ensure localization and simultaneous support of both the Uzbek and Russian languages ​​(initially, the project did not have a multinational search)

Dealing with localization challenges for the marketplace

We have been working on a service that enables a search index for products, categories, offers, and orders. The search index is organized through Elasticsearch.

The main challenge of the ecommerce localization was that we actually needed to localize two parts. The first was the content of the site itself, the non-changeable parts — all kinds of labels on buttons (such as confirm and cancel) and all kinds of names for different parts of the site (contact, search, selection, and so on).

All this had to be translated into Uzbek, while retaining the Russian language and adding English. So we had to manage the marketplace language settings and find a way to switch between the languages and show each user the correct website version, based on their IP address.

The second part, the names of the goods, had to be localized as well. Previously, they actually kept only names in Russian language. But when these headings were processed for indexes or a search query was processed, the rules for both Russian and English were used, since a lot of English words are used in Russian, including those typed in Latin. So now the service shows titles in two languages or even in three.

Another challenge when localizing content was that words in the Uzbek language can be written in both Latin and Cyrillic. So, in the search index, we first split these fields into two parts or double-fields. Then we connected two dictionaries (an Uzbek dictionary in Latin and an Uzbek dictionary in Cyrillic) to convert words into a standard form. This makes it possible to process the request correctly, in whatever variant it is entered.

Handling synonyms in a search engine

When there are so many complex language variants, the so-called synonym tables are very widely used to ensure that words that do not correspond to any language rules can still be reduced to some kind of canonical form.

Let's say we have the name Apple. We need to make it clear that we are talking about Apple the company and not about an apple as a fruit. And it’s important to make sure that the word is treated the same way in both the search query and the product title. To do this, there are tables of synonyms that can be edited from time to time. You can also evaluate the results of the change through statistical mechanisms to make sure that, for example, this change gave you an increase in search results, and that change gave you a decrease. So these files must be constantly updated.

The project did not have a system that would allow the simultaneous use and automatic updating of one such file across all servers and virtual machines that process this data. So we developed a system by connecting a virtualized hard disk through S3 cloud storage. This allowed us to organize the storage of synonyms into one single copy that is visible to all services that need it.

Technology stack

• Microservices are written in Go.
• PostgreSQL is used as a database for storing products, categories, and other mutable entities.
• Kafka for data transfer.
• Debezium to track changes in databases, write changes to Kafka.
• Automation and deployment tools: Helm, Terragrunt, Terraform, CircleCI, Drone for CI/CD testing.
• Settings are passed through environment variables (not configmaps).
• Kubernetes, S3 cloud storage.

Reliance on open-source

When we needed dictionaries for the local keyword search to reduce and normalize words in the Uzbek language, we had to either write our own solution or find a ready-made one. We found a nice Github solution, but the snag was, while the English version of the dictionary connected immediately, the Russian version did not. So it always crashed with an error.

We had to determine what was wrong there and fix the file so that it conformed to the error-free dictionary rules. So we figured out how to fix the problem and sent a pull request to the project. Our pull request was accepted and now everything works fine.

The conclusion

In addition to the main tasks already discussed, we also assisted with a number of routine tasks. For example, the repository had to be designed in a specific way that would allow it to receive updates from the original system, which is constantly evolving. Everything done on the old system, the new one needed to inherit. But, at the same time, changes made to the new system needed to stay there and not be sent to the old system.

In the final stages of the project, we helped with determining how well it all turned out — testing through the frontend and through special mechanisms that allow you to bombard access points with requests and compare the results.

We were thrilled to take part in the development and launch of such a promising project for Technokratos’s client, and we are sure that the marketplace localization will generate a large influx of new users. If you want to localize an ecommerce website or you need help developing a marketplace, send us a message using the form below, and we will contact you to discuss your project!