Peak Detection is a challenging step in many applications. Read and learn how to accurately detect peaks and valleys in 1D vectors and 2D arrays (images). — Our human brain is excellent in peak detection in relation to its context. What seems an easy task by eye can be a challenging task to automate by machines. In general, peaks and valleys indicate (significant) events such as sudden increases or decreases in price/volume, or sharp rises in demand…

The chances of successfully delivering data science projects are greatest when you keep learning, but it’s not always clear what to focus on — Many great developments in data science have been made in the last decade but despite these achievements, many projects never see the light of day. As data scientists we must not only show strong technical skills but also understand the business context, effectively communicate with stakeholders, and translate their questions…

Knowing the underlying data distribution is an essential step for data modeling and has many applications, such as anomaly detection, synthetic data creation, and data compression. — Knowing the underlying (probability) distribution of your data has many modeling advantages. The easiest manner to determine the underlying distribution is by visually inspecting the random variable(s) using a histogram. With the candidate distribution, various plots can be created such as the Probability Distribution Function plot (PDF/CDF), and the QQ…

Create interactive, stand-alone, and visually attractive charts that are built on the graphics of d3 javascript (d3js) but configurable with Python. — Python has become one of the most popular programming languages to analyze and visualize your data. Visualizing can be the key to success in projects because it can reveal hidden insights in the data, and improve understanding. The best way to understand and explain the data is by making it…

Detecting multicollinearity in data sets is an important step but also challenging. I will demonstrate how to detect variables with similar behavior in mixed data sets and how to deeper examine the relationships with interactive charts. — Understanding the strength of relationships between variables in a data set is important because variables with statistically similar behavior can affect the reliability of models. To remove the so-called multicollinearity we can use correlation measures for continuous variables. However, when we also have categorical variables and thus mixed data sets…

Machine learning solutions take an important place in our lives. It is not only about performance anymore but also about responsibility. — In the last decades, many AI projects focused on model efficiency and performance. Results are documented in scientific articles, and the best-performing models are deployed in organizations. Now it is the time to put another important part into our AI systems; responsibility. The algorithms are here to stay and nowadays…

Beautiful visualizations are great but to maximize the interpretability, you need to choose a chart carefully. — Visualization is an important part of data analysis as it can transform data into insights and help you with storytelling. In this blog post, I will focus on Network charts, Heatmaps, and Sankey charts. These charts have the same input, but we should keep in mind that they are designed…

Learn to effectively optimize hyperparameters, and prevent creating overtrained models for XGBoost, CatBoost, and LightBoost — Gradient boosting techniques such as XGBoost, CatBoost, and LightBoost has gained much popularity in recent years for both classification and regression tasks. An important part of the process is the tuning of hyperparameters to gain the best model performance. The key is to optimize the hyperparameter search space together with…

A tutorial on how to create a landscape using categorical data, and perform unsupervised analysis for deeper insights — The data science field is under constant development for which new roles and functions are created. The traditional data science role is evolving into tens of new roles, from Data Engineer, ML Engineer, Product Data Analyst, Research Scientist, Cloud Data Engineer, and many more. In this blog, we will load…