In the fall of 2017, ZAI was tasked with a project to search the Internet for violations of Government regulatory guidelines. Information from searched websites was to be collected, properly sorted, made available for comprehensive evaluation, and serve as the basis for recommended enforcement actions. The project posed several technical and operational challenges, including the need to crawl and mine a vast amount of data within a very short timeframe; organize hundreds of metadata elements for each item; append historic data into a repository for trending analysis; and enable our team to review, analyze, and report on the findings quickly and efficiently.
To address these challenges, ZAI developed an IT platform comprising a robust and modular database, search algorithm, and web crawler. The platform serves as a workflow and workload management system, automatically populating and prioritizing the harvested websites’ metadata into the database. This allows our team to focus on the most urgent items first, as well as evaluate the findings efficiently and recommend appropriate enforcement actions. Results of our data mining and analysis, including historic and current trends, are reported to the Government. ZAI received commendations for implementing its solution in a short timeframe and for producing high-quality results. By designing the platform to be modular, extensible, scalable, and secure, ZAI was able to customize it quickly and implement it on other projects.
