Data Science and Databases

The Hadoop Distributed File System (HDFS) is a scalable, open-source solution for storing and processing large volumes of data. With its built-in replication and resilience to disk failures, HDFS is an ideal system for storing and processing data for analytics.

In this step-by-step tutorial, Toptal Database Developer Dallas H. Snider details how to migrate existing data from a PostgreSQL database into the more efficient HDFS.

Genome data is one of the most widely analyzed datasets in the realm of Bioinformatics. The SciPy stack offers a suite of popular Python packages designed for numerical computing, data transformation, analysis and visualization, which is ideal for many bioinformatic analysis needs.

In this tutorial, Toptal Software Engineer Zhuyi Xue walks us through some of the capabilities of the SciPy stack. He also answers some interesting questions about the human genome, including: How much of the genome is incomplete? How long is a typical gene?

As a language, R is strongly tied to data and is thus used mostly by statisticians and data scientists. Many who already use R for machine learning, though, are not aware that data munging can be done faster in R, meaning another tool is not required for that task.

In this article, Freelance Software Engineer Jan Gorecki explores tabular data transformations and introduces us to one of the fastest open-source data wrangling tools available.

Ever tried to create a JSON data structure that includes entities with bidirectional relationships? If you have, you know that this often results in errors or exceptions being thrown.

In this article, Toptal Freelance Software Engineer Nirmel Murtic provides a robust working approach to avoiding these errors when creating JSON structures that included entities with bidirectional (i.e., circular) relationships.

More than 60 percent of trading activities with different assets rely on automated trading and machine learning instead of human traders. Today, specialized programs based on particular algorithms and learned patterns automatically buy and sell assets in various markets, with a goal to achieve a positive return in the long run.

In this article, Toptal Freelance Data Scientist Andrea Nalon explains how to predict, using machine learning and Python, which trade should be made next on the S&P 500 to get a positive gain.

Clustering algorithms remain an important part of unsupervised learning and machine learning workflows. These algorithms help uncover patterns in unlabeled data, but each comes with different strengths, trade-offs, and ideal use cases.

In this article, Toptal Freelance Software Engineer Lovro Iliassich explores clustering approaches ranging from K-Means and EM clustering to Affinity Propagation and DBSCAN.

When dealing with data analysis, most companies rely on MS Excel or Google Sheets, but dealing with data presented this way isn’t very eye-catching or intuitive. It’s once you add visualizations to this data that things become a little easier to manage. That’s the topic of today’s tutorial by our guest author from Adobe, Rohit Boggarapu. Join us as he guides us though the process of making interactive drill-down charts using jQuery and FusionCharts.

Database tuning can be an incredibly difficult task, particularly when working with large-scale data where even the most minor change can have a dramatic (positive or negative) impact on performance.

In mid-sized and large companies, most database tuning will be handled by a Database Administrator (DBA). But there are plenty of developers who have to perform DBA-like tasks; meanwhile, DBAs often struggle to work well with developers.

In this article, learn database tuning tips and how developers and DBAs can work together effectively.

Today, a massive amount of data is available in the form of networks or graphs. For example, the World Wide Web, with its web pages and hyperlinks, social networks, semantic networks, biological networks, citation networks for scientific literature, and so on.

A tree is a special type of graph, and is naturally suited to represent many types of data. The analysis of trees is an important field in computer and data science. In this article, we will look at the analysis of the link structure in trees. In particular, we will focus on tree kernels, a method for comparing tree graphs to each other, allowing us to get quantifiable measurements of their similarities or differences. This an important process for many modern applications such as classification and data analysis.

Hackathons often inspire engineers to create amazing software. By blending various technologies together, really useful and often fun projects can be realized in a short period of time.

In this article, Toptal engineer Radek Ostrowski shares his experience participating in the IBM Sparkathon, and walks us through how he elegantly combined the power of Apache Spark and Docker in IBM Bluemix to build a weather app.

Machine Learning, in computing, is where art meets science. Perfecting a machine learning tool is a lot about understanding data and choosing the right algorithm. But why choose one algorithm when you can choose many and make them all work to achieve one thing: improved results.

In this article, Toptal engineer Ivan Voras provides a useful overview and comparison of multi-processing network server models, with the goal being to take some of the mystery out of writing high performance networking code. The article is intended for “system programmers”, i.e., back-end developers who will work with the low-level details of their applications, implementing network server code.

Developers often work on only one machine, and have their whole development environment on that machine. Testing database replication before deploying changes in this kind of a development environment can be a challenging task.

In this article, Toptal engineer Ivan Bojovic guides us through a step-by-step tutorial on how to implement MySQL master-slave replication on one machine.

Developers sometimes run into tasks that require access to email mailboxes. In most cases, this is done using the Internet Message Access Protocol, or IMAP. As a PHP developer, I first turned to PHP’s built in IMAP library, but this library is buggy and impossible to debug or modify. So today we will create a working IMAP email client from the ground up using PHP. We will also see how to use Gmail’s special commands.

Image processing algorithms are often very resource intensive due to fact that they process pixels on an image one at a time and often requires multiple passes. Successive Mean Quantization Transform (SMQT) is one such resource intensive algorithm that can process images taken in low-light conditions and reveal details from dark regions of the image.

In this article, Toptal engineer Daniel Angel Munoz Trejo gives us some insight into how the SMQT algorithm works and walks us through a clever optimization technique to make the algorithm a viable option for handheld devices.

When Mike Bostock created D3.js, he introduced a tried and true reusable charts pattern for implementing the same chart in any number of selections. However, the limitations of this pattern are realized once the chart is initialized. In this article, Toptal engineer Rob Moore presents a revised reusable charts pattern that leverages the full power of D3.js.