Overcoming data collection challenges to build an effective influencer tracker

The Players advertising agency has been doing influencer marketing since 2014. They come up with and implement projects of varying complexity through bloggers on Instagram, YouTube, TikTok, VKontakte, Telegram, podcasts and other platforms. They execute projects ranging from global strategy and creative platforms to video production and posting. In their six years of existence on the market, Players has implemented more than 500 projects.

They specialize in blogger projects and provide services like integrated influencer marketing, influencer marketing strategies, creative concepts, media planning and influencer buying, consulting, video production and streaming, and podcast production. The agency’s clients include companies such as Nestle, Danone, Colgate, Xiaomi, McDonald’s, Samsung, Disney, and many more. They have the ability to offer such a wide range of services and projects thanks to our joint work on the creation of a system for tracking and selecting influencers. 

The challenge

In 2018, the Players agency approached Evrone for help developing an influencer marketing automation software. When they begin working with a new client, Players analyzes the target audience, indicators, and user profile, then finds influencers who match the client, within these parameters, in order to optimize the retention rate. Before contacting us for help, they used Excel spreadsheets as a database, and they had to manually search for influencers and their accounts on social networks and Youtube.

Their hypothesis was that everyone knows how to find well-known influencers, however, relevant influencers must have at least 20,000 subscribers to be useful for advertising. These bloggers can already be considered promising, and if you find the right approach and set up, for example, a one-year contract with them, over that year they can gain 1 million subscribers, and the client's audience will grow with them.

The problem was that users with an audience of 20-30 thousand subscribers could not easily be located manually. For example, Youtube does not provide the ability to search users by the number of subscribers.

The client needed us to build a marketing platform to automate the process of searching for and selecting influencers. This would allow them to find relevant influencers and initiate contracts with them before everyone else, giving their advertising a larger reach.

The solution

We started by writing technical specifications for the influencer search tool. In 6 months, we helped the client develop the MVP of a parsing and scraping system for Instagram, Facebook, and Youtube.

Since the client did not need a publicly available site, the core of our solution was an admin panel containing lists of influencers designed and developed by us. There were also background workers to grab the statistics of influencers from social networks, using the API, when possible, and scraping. Admin fields, comments, project lists, and prices could also be added manually.

This was a difficult task, and it took a long time to develop each parser. Each of these services hinder the collection of data in every possible way. They all give out a small piece of data, which is not enough to work with and is difficult to utilize. There is a lot of data that is not in the open API of social networks, and these sites also set limits on requests to the API. For example, Youtube has an API, but does not allow all data to be displayed. The agency needed to be able to parse statistics and monitor how their bloggers develop.

We developed the system in Ruby on Rails, and used React on the frontend. In addition, we used the Luminati VPN system, which is the world's largest proxy service, with a residential proxy network of 72 million IPs worldwide and a proxy management interface for zero coding.

Data scraping social networks, like Facebook, Instagram, and Youtube, is difficult, since they are very smart services constantly improving the blocking system. They block multiple requests, so you always have to use new IP addresses to bypass the blocking. The VPN system had to be used to connect from a new region, so that we would not be tracked and banned.

We also developed the design of the admin panel, drew graphics, and added elements on the frontend. You can go to the admin panel and see statistics to determine, for example, whether the blogger had a temporary surge in activity because of just one video or if they have actually had systematic growth and will likely continue to grow.

Overcoming limitations

In recent years, many companies have experienced data leaks, leading social networks and other services to tighten their security. As a result, businesses that rely on data from these sites are left wondering, “How do you scrape Facebook data in 2021?” While social media crawling and scraping are undeniably more difficult now, we learned several tricks from our time working on this project.

In 2018, there were problems with data leaks from Facebook, and Mark Zuckerberg was called in for a Senate hearing. At that time, we had already integrated the system with Facebook, and Facebook suddenly turned off many existing ways to collect data from it. Therefore, we had to modify the parsers to allow the system to scrape data from Facebook.

In addition, Instagram made it difficult to work with its API, so we made scrapers, robots that pretended to be people in the browser, walking through pages and parsing and collecting data. All of this happened shortly before the project was completed. It took a lot of time and work to make the changes, but we succeeded in creating an efficient MarTech platform.

Conclusion

We developed an influencer marketing software platform for the client that helped them to move from a huge number of Excel spreadsheets to one convenient dashboard. The client was pleased that we developed exactly what they needed to manage and expand their business.

The advertising technology solution we developed is the only one on the market, and the client still uses it to find influencers to work with. Since the initial development, Players has contacted us for additional modifications to further refine the product to meet their changing needs.

 If you need to develop a parsing system for your project, leave a request through the form below, and we will contact you as soon as possible to discuss how we can help.