The chances of successfully delivering data science projects are greatest when you keep learning, but it’s not always clear what to focus on — Many great developments in data science have been made in the last decade but despite these achievements, many projects never see the light of day. As data scientists we must not only show strong technical skills but also understand the business context, effectively communicate with stakeholders, and translate their questions…

Knowing the underlying data distribution is an essential step for data modeling and has many applications, such as anomaly detection, synthetic data creation, and data compression. — Knowing the underlying (probability) distribution of your data has many modeling advantages. The easiest manner to determine the underlying distribution is by visually inspecting the random variable(s) using a histogram. With the candidate distribution, various plots can be created such as the Probability Distribution Function plot (PDF/CDF), and the QQ…

Create interactive, stand-alone, and visually attractive charts that are built on the graphics of d3 javascript (d3js) but configurable with Python. — Python has become one of the most popular programming languages to analyze and visualize your data. Visualizing can be the key to success in projects because it can reveal hidden insights in the data, and improve understanding. The best way to understand and explain the data is by making it…

Machine learning solutions take an important place in our lives. It is not only about performance anymore but also about responsibility. — In the last decades, many AI projects focused on model efficiency and performance. Results are documented in scientific articles, and the best-performing models are deployed in organizations. Now it is the time to put another important part into our AI systems; responsibility. The algorithms are here to stay and nowadays…

Beautiful visualizations are great but to maximize the interpretability, you need to choose a chart carefully. — Visualization is an important part of data analysis as it can transform data into insights and help you with storytelling. In this blog post, I will focus on Network charts, Heatmaps, and Sankey charts. These charts have the same input, but we should keep in mind that they are designed…

Learn to effectively optimize hyperparameters, and prevent creating overtrained models for XGBoost, CatBoost, and LightBoost — Gradient boosting techniques such as XGBoost, CatBoost, and LightBoost has gained much popularity in recent years for both classification and regression tasks. An important part of the process is the tuning of hyperparameters to gain the best model performance. The key is to optimize the hyperparameter search space together with…

A tutorial on how to create a landscape using categorical data, and perform unsupervised analysis for deeper insights — The data science field is under constant development for which new roles and functions are created. The traditional data science role is evolving into tens of new roles, from Data Engineer, ML Engineer, Product Data Analyst, Research Scientist, Cloud Data Engineer, and many more. In this blog, we will load…

Reveal the hidden causal variables in your data set by using the best-suited Bayesian causal inference library: a comparison with hands-on examples of five popular libraries. — Understanding the causal effect of variables in systems or processes is very valuable. There are a number of Python libraries that can assist in determining causal relationships. I will compare five popular causal inference libraries in their functionality, ease of use, and flexibility. Each is accompanied by hands-on examples. The…

Learn how to quantitatively detect which features drive the formation of the clusters — Cluster analysis is a great technique for identifying groups with similar patterns. However, once clusters are formed, it can remain challenging to determine the driving features behind the clusters. But this step is crucial to reveal valuable insights that may have been missed before and can be used for decision-making…