A system for automating the processes of identifying significant events in the media environment

A system for automating the processes of identifying significant events in the media environment, generating dynamic reports and system management of solutions based on identified events.

Client:

A full-cycle market research agency.

Tasks:

To simplify the collection and subsequent analysis of reactions in social networks to certain information occasions and digitize the results obtained.

The system provides:

• The feature to add publications, search for added publications, view System user actions as a part of the publication function.
• The feature to view and edit information events, their distribution in the media, social networks and messenger.
• Monitoring of reactions in social networks and mass media to incidents and messages published to them.
• The feature to create and search objects made by a system user in a text.
• The feature to create, edit and view the results of dynamic reports with the function to browse through collections of TOP articles from the media and social networks, news on specified topics, news with quotes from specified persons, a list of news.
• Collecting and storing articles from online media sources, social networks and messengers.

Tasks:

The social network monitoring solution was developed:

To process data from all major Russian and international social networks, as well as from more than 150,000 local blogs and forums.

To provide analytical tools and aggregated statistics on users and content

One of the largest software outsourcing providers in Central and Eastern Europe has spent almost 2 years and a mind-boggling $2 million to release a product that can’t even handle more than 5 users and has a minimum indexing requirement of 100,000 messages per day. Investors were enraged and demanded that our team take over the management.

Solution:

To avoid after-troubles, the client decided to remade the entire architecture. Our team began deploying 10 prototypes, which had been thoroughly studied with an emphasis on performance and scalability. It took us 2 months to develop the final design of the system that meets the requirements of the NFR:

Data exchange has been moved to a queue based on RabbitMQ.

Instead of one large database for all users and reports, the team developed a small one with a chain of tables for each user, which made it possible to better manage complex reports and reduce the cost of scaling on small instances.

Scrum, 100% test automation and continuous integration allowed us to conduct weekly deployments and beta testing every 2-3 weeks.

Results:

Having saved up to 50% of the budget, our team has provided a scalable solution that can ensure further growth of the client base and the volume of processed social data, at least for the next few years.

Project implementation period:

8 months

Team (roles)

MP, Architect, Analyst, TeamLead, Developer 1, Developer 2, Developer 3, UI/UX Designer, Tester 1, Tester 2, Technical Support Specialist.

Technology stack.

NET framework, js, Server OS based on Windows and Linux, Database powered by MSSQL, MongoDB, Apache Cassandra, IIS and Nginx-based web servers, Docker-based containerizing, RabbitMQ-based message broker, Logging based on ELK, Monitoring based on Grafana / Prometheus and Zabbix.