Big Data Analytics: Wisdom or Folly!

There are hosts of buzzwords in today’s data-centric world, and especially in digital and print media. We encounter data in every walks of life, and for analytically and objectively-minded people, data is everything. However, making sense of the data and extracting meaningful information from it may not be an easy task. We come across buzzwords such as big data, high dimensional data, data science, and open data without a proper definition of such words. The rapid growth in the size and scope of data sets in a host of disciplines has created a need for innovative statistical strategies analyzing such data. For example, many private and public agencies are using sophisticated data mining strategies and/or big data analytics to reveal patterns based on collected information. Some examples of big data that have prompted demand are digital marketing, customer service standards, gene expression arrays, social network modeling, clinical, genetics and phenotypic data.

The need for novel statistical strategies to analyze such data sets is pressing. This talk focuses on the development of statistical and computational strategies for a sparse regression model in the presence of mixed signals. The existing estimation methods have often ignored contributions from weak signals. However, in real scenario many predictors altogether provide useful information for prediction, although the amount of such useful information in a single predictor might be modest. The search for such signals, sometimes called networks or pathways, is for instance an important topic for those working on personalized medicine. We discuss a new “post selection shrinkage estimation strategy” that takes into account the joint impact of both strong and weak signals to improve the prediction accuracy and opens pathways for further research in such scenarios.