Let's learn about Big Data via these 500 free blog posts. They are ordered by HackerNoon reader engagement data. Visit the /Learn or LearnRepo.com to find the most read blog posts about any technology.
Gather and organize and process insights from large datasets with new computer strategies and technologies
1. 13 Best Datasets for Power BI Practice
In 2022, Gartner named Microsoft Power BI the Business Intelligence and Analytics Platforms leader. These are the 13 Best Datasets for Power BI Practice.
2. Navigating Big Data's Potential and Privacy in Modern Medicine
How can we harness big data to advance healthcare while protecting sensitive patient information? This article explores and answers that question.
3. The Top 16 Types of Charts in Data Visualization That You'll Use
In the era of information explosion, more and more data piles up. However, these dense data are unfocused and less readable. So we need data visualization to help data to be easily understood and accepted. By contrast, visualization is more intuitive and meaningful, and it is very important to use appropriate charts to visualize data.
4. Pyth and Auros are Bringing Real-Time High-Frequency Data to Blockchain Protocols
Auros, a company specialising in algorithmic trading and market making, and Pyth Network will provide access to high-frequency data in real-time.
5. An Intro to Resiliency, DHT, and Autonomous Economic Agents
According to the paper published by Lokman Rahmani et al., the S/Kademlia distributed hash table (DHT) used by the ACN is resilient against malicious attacks.
6. Advantages and Disadvantages of Big Data
Big data may seem like any other buzzword in business, but it’s important to understand how big data benefits a company and how it’s limited.
7. Top 10 JavaScript Charting Libraries for Every Data Visualization Need
There're numerous JavaScript charting libraries. To make your life easier, I decided to share my picks. Check out the best JS libraries for creating web charts!
8. 6 Biggest Limitations of Artificial Intelligence Technology
While the release of GPT-3 marks a significant milestone in the development of AI, the path forward is still obscure. There are still certain limitations to the technology today. Here are six of the major limitations facing data scientists today.
9. Top 10 Open Datasets for Linear Regression
On Hacker Noon, I will be sharing some of my best-performing machine learning articles. This listicle on datasets built for regression or linear regression tasks has been upvoted many times on Reddit and reshared dozens of times on various social media platforms. I hope Hacker Noon data scientists find it useful as well!
10. 16 SQL Techniques Every Beginner Needs to Know
This blog post explains the most intricate data warehouse SQL techniques in detail.
11. Crunching Large Datasets Made Fast and Easy: the Polars Library
Processing large data, e.g. for cleansing, aggregation or filtering is done blazingly fast with the Polars data frame library in python thanks to its design.
12. How Wikipedia Lost 3 Billion Organic Search Visits To Google in 2019
Since Wikipedia was founded in 2001, people worldwide rely on the online encyclopedia to expand their horizons and read information on just about anything. As true as that is today, however, the site’s traffic trends tell a very different story.
13. Python vs JavaScript: Main Differences, Performance Comparison, and Areas of Application
The complexity of modern web apps lies far beyond creating eye-catching user interfaces with countless elements. To enable lag-free experience and effortless scalability, it’s important to pay due attention to the architecture design, which can be pretty challenging. Under the hood of a full-featured online app, different frameworks and libraries can peacefully coexist with different programming languages used to build software. Since the equation may contain so many variables, it’s essential to master your knowledge of each potential system component to know when and why to use them.
14. A Deep Dive Into Amdahl’s Law and Gustafson’s Law
Discover in detail the background, theory, and usefulness of Amdahl's and Gustafson's laws. We also discuss the strong and weak scaling tests on a C++ CFD ...
15. Eliminating Difference Between Business Intelligence analysts, Data Analysts or Data Scientists 🚀
There was a time when the data analyst on the team was the person driving digitalization in an adventurous data quest...and then the engineers took over.
16. Hadoop Across Multiple Data Centers
Hadoop cluster across multiple data centers
17. 3 Best Hadoop Alternatives to Consider for Migration
In this article, we will discuss why Hadoop is losing popularity and what other options are available that could potentially replace it.
18. $DAG Will Do To Big Data What Bitcoin Did To Money
Hello, Dear reader! 🧑💻 Here I talk about the Constellation Network, Inc. Why I think the Constellation is one of the most amazing companies! Why they will steal the show and create and set the standard for future Cybersecurity for Big Data. I give arguments to which I paid more attention than to others, as possible clearly and briefly. Go!
19. The Best (and Worst) Punny Jokes Only Data Scientists Will Understand
For the first KDnuggets post on Hacker Noon, we bring you a lighter fare of very nerdy computer humor from the series of self-referential jokes started on Twitter earlier this week. Here are some of our favorites.
If you do understand all of the jokes, then you congratulate yourself on having excellent knowledge of Data Science and Machine Learning! If you have actually laughed at 2 or more jokes, then you have earned MS in Computer Humor! If you just smirked, you probably have a Ph.D. And I have a great joke about AGI, but it will be ready in 10 years.
Enjoy, and if you have more, add them in comments below!
Yann LeCun, @ylecun
20. How GPUs are Beginning to Displace Clusters for Big Data & Data Science
More recently on my data science journey I have been using a low grade consumer GPU (NVIDIA GeForce 1060) to accomplish things that were previously only realistically capable on a cluster - here is why I think this is the direction data science will go in the next 5 years.
21. Limit Cloud Data Costs with MinIO on Equinix
Companies who understand the cloud operating model and understand their workloads are voting with their feet.
22. Apache Druid, TiDB, ClickHouse, or Apache Doris? A Comparison of OLAP Tools
The OLAP experience of an automobile manufacturer.
23. Web Scraping con Python: Guía Paso a Paso
La necesidad de extraer datos de sitios web está aumentando. Cuando realizamos proyectos relacionados con datos, como el monitoreo de precios, análisis de negocios o agregador de noticias, siempre tendremos que registrar los datos de los sitios web. Sin embargo, copiar y pegar datos línea por línea ha quedado desactualizado. En este artículo, le enseñaremos cómo convertirse en un "experto" en la extracción de datos de sitios web, que consiste en hacer web scraping con python.
24. The Importance of Hypothesis Testing
Hypothesis tests are significant for evaluating answers to questions concerning samples of data.
25. Automated Data Replication From AWS S3 To Microsoft Azure Storage Made Easy
It may be a requirement of your business to move a good amount of data periodically from one public cloud to another. More specifically, you may face mandates requiring a multi-cloud solution. This article covers one approach to automate data replication from AWS S3 Bucket to Microsoft Azure Blob Storage container using Amazon S3 Inventory, Amazon S3 Batch Operations, Fargate, and AzCopy.
26. 6 Database Migration Tools For Complete Data Integrity & More
Database migrations are driven by benefits like lower costs, better features, and the ability to scale. However, the security of data is essential.
27. A Comprehensive Guide to Building DolphinScheduler 3.2.0 Production-Grade Cluster Deployment
In version 3.2.0, DolphinScheduler introduces a series of new features and improvements, significantly enhancing its stability.
28. Web Scraping con Python: Guía Paso a Paso
La necesidad de extraer datos de sitios web está aumentando. Cuando realizamos proyectos relacionados con datos, como el monitoreo de precios, análisis de negocios o agregador de noticias, siempre tendremos que registrar los datos de los sitios web. Sin embargo, copiar y pegar datos línea por línea ha quedado desactualizado. En este artículo, le enseñaremos cómo convertirse en un "experto" en la extracción de datos de sitios web, que consiste en hacer web scraping con python.
29. How to Scrape Data from Google Maps
Want to scrape data from Google Maps? This tutorial shows you how to do it.
30. Big Brother Meets Black Mirror in the Middle Kingdom
Imagine a world where everything you ever do or say is watched and rated by invisible eyes.
31. Busting AI Myths: "You Need Tons of Data for Machine Learning"
Leading researchers like Karl Friston describe AI as "active inference" —creating computational statistical models that minimize prediction-error. The human brain operates much the same way, also learning from data. A common argument goes:
32. Data Engineering: An Interview with Meta Engineer Leonid Chashnikov
As we sit down for this exclusive interview, Leonid offers a rare glimpse into the intricate process of weaving the digital fabric that shapes our lives.
33. Migrating From Hadoop Without Rip and Replace Is Possible — Here's How
Here's how to migrate from Hadoop without the need to completely overhaul your existing systems.
34. How to Start with Web Scraping and Why You Don't Need to Code
Collecting data from the web can be the core of data science. In this article, we'll see how to start with scraping with or without having to write code.
35. Writing a Scraping Bot with Python and Selenium
Learning how to use Selenium and Python to interact with websites to get the data you need.
36. AI Meets Ethics: Navigating Bias and Fairness in Data Science Models
Explore a product developer's journey in tackling AI bias and fairness. Learn how ethical considerations shape AI design, ensuring technology benefits everyone.
37. Use Amazon Personalize & Data in the Raw for Real-Time Recommendations:
Start capturing website user data in 5 minutes or less with no developer resources or coding experience needed.
38. Writing Idempotent Code: A Guide
Idempotence, in programming and mathematics, is a property of some operations such that no matter how many times you execute them, you achieve the same result.
39. Performance Benchmark: Apache Spark on DataProc Vs. Google BigQuery
When it comes to Big Data infrastructure on Google Cloud Platform , the most popular choices Data architects need to consider today are Google BigQuery – A serverless, highly scalable and cost-effective cloud data warehouse, Apache Beam based Cloud Dataflow and Dataproc – a fully managed cloud service for running Apache Spark and Apache Hadoop clusters in a simpler, more cost-efficient way.
40. Uploading a 1 Million Row CSV File to the Backend in 10 Seconds
Uploading 1 million row size large CSV to mongoDB using nodejs stream
41. Top 8 Best Qlik Sense Extensions
Qlik Sense is powerful data visualization and BI software. But sometimes its functions are not enough. Meet the best Qlik Sense extensions to do more with data!
42. Top 5 Big Data Frameworks Developer Should Learn
These are the best Big Data Frameworks developers can learn. It includes Apache Hadoop, Apache Spark, Apache Flink, Apache Storm, and Apache Hive
43. How to Think Like a Data Scientist or Data Analyst
44. Elevating Enterprise LLMs with Retrieval-Augmented Generation (RAG) and Vector Database Integration
Dive into the challenges faced by LLMs and the transformative solutions offered by Retrieval-Augmented Generation (RAG) and vector databases.
45. Why Big Data is Big Business: The Netflix Example
Take a look at the following chart:
46. Predictive Analytics for Maintenance Events
The predictive analytics machine learning model worked well to provide alerts before the engine values went beyond thresholds avoiding expensive repair cost.
47. Is Web Scraping Stealing?
Web scraping is a super helpful tool not just to make money but also to reveal injustices hidden in plain sight, or to call Russians to talk about the war
48. Build vs Buy: What We Learned by Implementing a Data Catalog
Why we chose to finally buy a unified data workspace (Atlan), after spending 1.5 years building our own internal solution with Amundsen and Atlas
49. Best Practices For Apache Kafka Configuration
Having worked with Kafka for more than two years now, there are two configs whose interaction I've seen be ubiquitously confused.
50. Sustainable Computing beyond the Cloud
Extreme increases in data streams are expanding the cloud's carbon footprint; a sustainable alternative to Cloud dependence has been developed.
51. Efficient Data Storage for Rapid Analysis and Visualization
In this article, I want to share one of the ways that big data can be stored and used for analysis.
52. How AI Is Transforming Your Smartphone
The tech industry and the world are relying on artificial intelligence to solve big problems such as cybersecurity, healthcare and sustainability.
53. The Top Big Data Consulting Firms
Thanks to big data, today an organization can quickly obtain the necessary information from an unordered data set and deploy it effectively. The growing popularity of big data analytics has led to a significant increase in the number of companies providing big data solutions and related services.
54. How to Fix Data Skew in Apache Spark with the Salting Technique
Learn how to fix data skew in Apache Spark using the salting technique for improved performance and balanced partitions in Scala and PySpark.
55. How Big Data Will Impact the Accounting Industry
If I say that we have officially entered into the age of data, it would not be farfetched. According to the World Economic Forum, the total data produced in a day would reach to 44 zettabytes in 2020.
56. Effective Management of Data Sources in Machine Learning
Efficiently handling data sources is crucial for effective machine learning. Strategies include batch annotation, active learning, tracking annotator quality
57. Zero-Downtime Splunk Migration at inDrive: From Bare Metal to AWS SmartStore
How to migrate Splunk to AWS SmartStore with zero downtime using hybrid architecture, S3 storage, and multi-cluster search.
58. Using Rate Limiting Algorithms for Data Processing Pipelines
You may have already heard of rate limiting associated with REST API consumption. In this article I’ll show you a more complex use of this component...
59. 3 Top Resources To Learn About Apache Kafka
Top 3 books and tutorials on Apache Kafka
60. Retraining Machine Learning Model Approaches
Retraining Machine Learning Model, Model Drift, Different ways to identify model drift, Performance Degradation
61. Top 10 Best Web Scraper And Data Scraping Tools
Data extraction has many forms and can be complicated. From Preventing your IP from getting banned to bypassing the captchas, to parsing the source correctly, headerless chrome for javascript rendering, data cleaning, and then generating the data in a usable format, there is a lot of effort that goes in. I have been scraping data from the web for over 8 years. We used web scraping for tracking the prices of other hotel booking vendors. So, when our competitor lowers his prices we get a notification to lower our prices to from our cron web scrapers.
62. The Essential Architectures For Every Data Scientist and Big Data Engineer
Comprehensive List of Feature Store Architectures for Data Scientists and Big Data Professionals
63. Unraveling the Maze of Large JSON Files: Tips and Tools for Local JSON Parsing
Discover how a backend developer overcomes obstacles in processing large JSON log files.
64. 6 Biggest Differences Between Airbyte And Singer
We’ve been asked if Airbyte was being built on top of Singer. Even though we loved the initial mission they had, that won’t be the case. Aibyte's data protocol will be compatible with Singer’s, so that you can easily integrate and use Singer’s taps, but our protocol will differ in many ways from theirs.
65. Pilosa: A Scalable High Performance Bitmap Database Index
Big data is a big problem, at least getting anything useful out of it. Every day there is about three quintillion (the next step up is sextillion or one zettabyte) bytes of data created and only about 20% of it is structured and available to easily process. Nearly all useful processing that is done relies on a philosophy that is little changed from the green bar reports we were generating during the night shift and handing out up till the turn of the century. The whole map/reduce process is overnight batch processing, you aren’t working on live data, you are working on a snapshot, which might be fine for some companies, but for others, they need to be able to make decisions on high-velocity inbound data in near/real time.
66. Digging into Postgres's Lesser Known Features
Postgres Handles More than You Think
67. What the Heck is GlareDB?
Learn more about GlareDB and how it can fit in your data stack
68. 10 Ways to Optimize Your Database
Take these 10 steps to optimize your database.
69. Semi-Supervised Machine Learning Algorithms
70. The Future of Gaming: Leveraging Data Engineering to Revolutionize Player Experience
Explore how data engineering revolutionizes gaming with AI, AR/VR, blockchain, and more, enabling immersive experiences and shaping the industry's future.
71. Covid-19: Analysing The Spread Across Populations
A large portion of mild and asymptomatic cases may go unreported. The data will never be perfect, the true cases are likely much larger as the testing frequency and effectiveness vary in different regions.
72. The Role of Big Data in Developing New Medicines
Drug development is one of the most crucial — and time-consuming — processes in medicine. Here's how big data can help.
73. Dataism: Idea or Ideology?
Dataism suggests that the entire universe can be interpreted as data flows and that all phenomena, including human behaviour, can be reduced to data processes.
74. Digging Into Amazon's Privacy Policy
Amazon has developed a reputation for delivering some of the lowest prices for all types of products, and one of the best delivery systems in the world. Part of what makes this possible is Amazon’s extensive use of people’s data. We’re taking a look at which information Amazon collects and how it collects that information.
75. Automating the Automation: Can AI Fully Take Over the Data Scraping Process?
Can modern AI systems fully automate web data collection and analysis? Let’s delve deeper into ML and web scraping to see if this is more than just a new hype.
76. 5 Prominent Big Data Analytics Tools to Learn in 2020
Data, data and data. This seems to be what our world is swimming and immersing in. Why? The answer is simple: simply everything we use, such as mobile phones, and with it, all that it has, such as the social media, churn out unimaginable amounts of data.
77. Scale Your Data Pipelines with Airflow and Kubernetes
It doesn’t matter if you are running background tasks, preprocessing jobs or ML pipelines. Writing tasks is the easy part. The hard part is the orchestration— Managing dependencies among tasks, scheduling workflows and monitor their execution is tedious.
78. Probabilistic Data Structures And Algorithms In Big Data
Probabilistic data structures allow you to conquer the beast and give you an estimated view of some data characteristics
79. Containerization of Spark Python Using Kubernetes
Introduction
80. Advancing User Data Governance with Data Lineage
This article will discuss how data lineage can help in user data governance and explore how serverless technology can be incorporated to achieve better results.
81. Extraer Datos del Website a Excel Automáticamente
82. Graph Databases: Full Detailed Review
There are many ideas and considerations behind graph databases. This includes their use cases, advantages, and the trends behind this database model. There are also several real-world examples to dissect.
83. How this Web3 Project is Unlocking a Trillion-Dollar Data Economy with Data NFTs
Learn why data could become the most promising NFT utility that sets the foundation for a valuable trend: Data Finance (DataFi).
84. Top Industry Trends for AI Marketing
Companies that embrace AI will be able to test, learn, and iterate much faster, raising the competitive bar for learning.
85. Artificial Intelligence and Big Data
Artificial Intelligence and Big Data. These two terms seem to permeate the tech world in every possible way one can think of. Along with giant terms like Machine Learning, IoT, blockchain and related ones, AI and Big Data are set to dominate our world in the years ahead.
86. From Big Data to Personal Lives: This Is How AI-Powered Tools Will Help Today’s Professionals
“AI is everywhere around us. We are living with it every day, and we are loving it.”
87. Analyzing Data From U.S. Road Accidents With Data Visualization
In this article, we would be analyzing data related to US road accidents, which can be utilized to study accident-prone locations and influential factors.
88. The Evolution of Big Data And Web Scraping
As the CEO of a proxy service and data scraping solutions provider, I understand completely why global data breaches that appear on news headlines at times have given web scraping a terrible reputation and why so many people feel cynical about Big Data these days.
89. Why Data Privacy is Important for Users in the Web3 Ecosystem
Interview discussing why data privacy is important for users in the web3 ecosystem
90. How Vectors, RAG and Llama 3 Are Changing First-Party Data
In the battle for the best data, is first-party better? Not by itself, but it could be with vectors, frameworks like RAG, and open-source models
91. How to Fit an Elephant in a Spreadsheet
Discover a faster way to cluster massive datasets without sacrificing accuracy.
92. Hadoop for Hoops: Explore the Whole Ecosystem and to Know How It Really Works
Technological evolution has changed the landscape, everything which we feel and hear today is revolving around some of the modern technology. This technology involves Artificial Intelligence, big data, cloud computing, data science, and much more, which has changed the landscape to a great extent. To integrate this technology, many of the IT professionals are finding and implementing the trajectory of today's modern technologies.
93. Data Science Teams are Doing it Wrong: Putting Technology Ahead of People
Data Science and ML have become competitive differentiator for organizations across industries. But a large number of ML models fail to go into production. Why?
94. Migrate Data from S3 to Snowball
In this article, I will show you how to migrate data from S3 to Snowball.
95. How AI is Disrupting the Legacy Systems of the Airline Industry
Andre Americo, Director of Revenue Management at Azul Airlines, discusses the impact of AI on revenue management in the airline industry.
96. Top 6 CI/CD Practices for End-to-End Development Pipelines
Maximizing efficiency is about knowing how the data science puzzles fit together and then executing them.
97. How Are Smart Cities Made 'Smart': Top 6 Enabling Technologies
The ultimate goal of smart cities is to improve citizens’ quality of life, reduce the cost of living and attain a sustainable environment through technology.
98. AI, Big Data, Blockchain, and Edge: Welcome to 2020
99. Certify Your Data Assets to Avoid Treating Your Data Engineers Like Catalogs
Data trust starts and ends with communication. Here’s how best-in-class data teams are certifying tables as approved for use across their organization.
100. Introducing the Swahili News Dataset for Topic Classification
Swahili (also known as Kiswahili) is one of the most spoken languages in Africa. It is spoken by 100–150 million people across East Africa. Swahili is popularly used as a second language by people across the African continent and taught in schools and universities. In Tanzania, it is one of two national languages (the other is English).
101. 9 Best Data Integration Software in 2022
102. Data Preparation for Machine Learning: A Step-by-Step Guide
Many businesses assume that feeding large volumes of data into an ML engine is enough to generate accurate predictions.
103. A Deep Dive Into Facebook’s AI Transcoder
Just over a week, most of you would have heard that Facebooks AI research team (FAIR) developed a neural transcompiler, that converts code from high level programming language like C++, Python, Java, Cobol into another language using ‘unsupervised translation’ . The traditional approach had been to tokenize the source language and convert it into an Abstract Syntax Tree (AST) which the transcompiler would use to translate to the target language of choice, based on handwritten rules that define the translations, such that abstract or the context is not lost.
104. AI: From ZERO to H...aving A Lot of Questions (Part I)
People are just like a Swiss Army Knife, but we are born with no tools on it. Everything we learn might become a new tool. With enough tools, we can accomplish everything. With the right tools, we can accomplish it faster, better and enjoy the endorphin rush.
105. Hadoop Data Storage Explained
Explore how exactly distributed storage works in Hadoop? We have to characterize an essential node (known as NameNode) from one of the workers (DataNodes).
106. Data Playgrounds are The Cure for Slow and Inefficient DataOps
Companies struggle with their DataOps due to a flawed, code-centric, and linear workflow. To succeed, they must build data playgrounds, not mere pipelines.
107. The Noonification: Feature Optimization for Price Prediction (11/26/2023)
11/26/2023: Top 5 stories on the Hackernoon homepage!
108. Why Parallel Programming is a Game Changer
Discover why parallel programming is a game-changer today. Role of Moore's law in the past, why it's dead, and how multicore processors became inevitable!
109. Who Will Eventually Control Big Data in Web3?
Web 3 is loudly making rounds as a decentralized internet. How will this affect data control in general?
110. Top 10 On-Demand IT Certifications With Highest Pay: 2020 Edition
Information Technology (IT) certification can enrich your IT career and pave the way for a profitable way. As the demand for IT professionals increases, let's look at 10 high-paying certifications. The technology landscape is constantly changing and the demand for information technology certification is also getting higher. Popular areas of IT include networking, cloud computing, project management, and security. Eighty percent of IT professionals say certification is useful for careers and the challenge is to identify areas of interest. Let's take a look at the certifications that are most needed and the salaries that correspond to them.
111. Lessons for Improving Training Performance — Part 1
Part 1: Lower precision & larger batch size are standard now
112. Industry 4.0’s Ultimate Impact on Manufacturing Business
113. Building a Large-Scale Interactive SQL Query Engine with Open Source Software
This is a collaboration between Baolong Mao's team at JD.com and my team at Alluxio. The original article was published on Alluxio's blog. This article describes how JD built an interactive OLAP platform combining two open-source technologies: Presto and Alluxio.
114. Integrate Apache Doris Into Your Data Architecture: Real-time Data Warehousing
A whole-journey guide for financial users looking for fast data processing performance, data security, and high service availability with Apache Doris.
115. What Apple And Spotify Know About Me
Unsurprisingly, the data that our apps have collected about us is both impressive and concerning, though it can be very interesting to review and explore it.
116. Not data-driven: purpose-driven and data-assisted
117. Data Science Training and Data Science - Machine Learning With Python
The requirement for its stockpiling also grew as the world entered the period of huge information. The principle focal point of endeavors was on structure framework and answers for store information. When frameworks like Hadoop tackled the issue of capacity, preparing of this information turned into a challenge. Data science began assuming a crucial job to take care of this issue. Information Science is the fate of Artificial Intelligence as It can increase the value of your business.
118. Data Engineering Tools for Geospatial Data
Location-based information makes the field of geospatial analytics so popular today. Collecting useful data requires some unique tools covered in this blog.
119. Unveiling Causal Impact: From Theory to Practice
We will guide you through a specific dataset, demonstrating how to implement the library and interpret results.
120. 3 Easy Ways to Improve The Performance Of Your Python Code
I. Benchmark, benchmark, benchmark
121. How to Achieve Optimal Business Results with Public Web Data
Public web data unlocks many opportunities for businesses that can harness it. Here’s how to prepare for working with this type of data.
122. How to Get Started with Data Governance Best Practices
Long recognized as a must in the data-driven world, data governance has never been easy for big and tiny organizations alike.
123. How Big Data and AI Help People Make Smarter Investments
Big data, artificial intelligence, and machine learning are some of the hottest technologies out there. Well, machine learning has existed since the late 1950s, and big data got first coined in 2005. However, it is only in the last decade, or so that computer engineers, scientists, and corporations have tried widespread implementations of these technologies.
124. The Emerging Data Engineering Trends You Should Check Out In 2024
Integrating data engineering with AI has led to the popularity of modern data integration and the expertise required.
125. Behavioral Intent Prediction Is Coming. Are We Ready?
It can feel at times like we live in a science fiction future. We hold the whole of human knowledge in palm-sized devices that are constantly connected to the Internet. We speak to our computers and they respond with seemingly intelligent feedback.
126. A/B Testing was a Jerk, Until we Found the Replacement for Druid
The recipe for successful A/B testing is quick computation, no duplication, and no data loss. So, we used Apache Flink and Doris to build our data platform.
127. How to Build Machine Learning Algorithms that Actually Work
Applying machine learning models at scale in production can be hard. Here's the four biggest challenges data teams face and how to solve them.
128. 693 Stories To Learn About Data
Learn everything you need to know about Data via these 693 free HackerNoon stories.
129. Can Blockchain Technology Help with Our Growing Privacy Problems?
Since the Internet's introduction to the public from the academic world, privacy issues have existed. Blockchain technology may be able to change this.
130. Top 3 Benefits of Insurance Data Analytics
The Importance of data analytics and data-driven decisions across the board and in this case insurance data.
131. Unveiling the Architecture: Key Papers to Understand Distributed Systems!
Top papers on distributed systems; distributed system papers every software engineer should read.
132. SQL Queries: Why You Need SQL-Agnostic Parsing
No need to be an expert in thousands of combinations of SQL, data types, and databases to master SQL queries. A good SQL agnostic parser will take care of all.
133. How is Web Crawling Used in Data Science
No-Code tools for collecting data for your Data Science project
134. Database Vs Data Warehouse Vs Data Lake: A Simple Explanation
A data lake is totally different from a data warehouse in terms of structure and function. Here is a truly quick explanation of "Data Lake vs Data Warehouse".
135. The API to Bootstrap Your Flink Jobs Has Arrived
Apache Flink is one of the most versatile data streaming open-source solution that exists. It supports all the primary functions of a typical batch processing system such as SQL, Connectors to Hive, Group By, etc. while providing fault-tolerance and exactly-once semantics. Hence, you can create a multitude of push-based applications using it.
136. Breaking Down Data Silos: How Apache Doris Streamlines Customer Data Integration
Learn how Apache Doris breaks down data silos for insurance firms, streamlining customer data integration and boosting efficiency.
137. Docker Dev Workflow for Apache Spark
The benefits that come with using Docker containers are well known: they provide consistent and isolated environments so that applications can be deployed anywhere - locally, in dev / testing / prod environments, across all cloud providers, and on-premise - in a repeatable way.
138. Future of Marketing: How Data Science Predicts Consumer Behavior
Gradually, as the post-pandemic phase arrived, one thing that helped marketers predict their consumer behavior was Data Science.
139. How To Meaningfully Interpret COVID-19 Data
140. Data Will Never Be Clean But You Can Make it Useful
Understanding how to clean data is essential to ensure your data tells an accurate story
141. 7 Challenges in Marketing AI & Machine Learning Solutions
This article will help our readers to identify and understand the challenges faced by the AI development companies to market the AI & ML products.
142. An Intro to Web Scraping: What it is and How to Start
A quick introduction to web scraping, what it is, how it works, some pros and cons, and a few tools you can use to approach it
143. How to Think Like a Data Systems Engineer: The Questions That Save You Later
Learn how engineers think about reliability, scalability, and maintainability—by asking the right questions early.
144. A Beginner's Introduction to Database Backup Security
With more companies collecting customer data than ever, database backups are key.
145. How to Migrate from Airflow to Dolphinscheduler in Two Steps
Recently, Air2phin, a scheduling system migration tool, announced its open source. With Air2phin, users can migrate the scheduling system from Airflow to Apache
146. How We Use dbt (Client) In Our Data Team
Here is not really an article, but more some notes about how we use dbt in our team.
147. How Big Data and Computers Leveled Up India in the 1950s
When we think of computers, we think of the twenty-first century. But did you know that India started using them back in the 1950s?
148. Get Machine Learning Training Data Using The Lionbridge Method [A How-To Guide]
In the field of machine learning, training data preparation is one of the most important and time-consuming tasks. In fact, many data scientists claim that a large portion of data science is pre-processing and some studies have shown that the quality of your training data is more important than the type of algorithm you use.
149. Building the Next-Generation Data Lakehouse: 10X Performance
How to connect various data sources easily and ensure high query performance.
150. Compete on Data Analytics using Spring Cloud Data Flow
Data Driven
151. A Deep Dive Into SeaTunnel Metadata Caching
How does SeaTunnel Zeta handle 10k+ tasks?
152. Online Privacy is Not an Option: It's a Necessity
153. Benchmarking Database Performance: Key OLTP and OLAP Tools for System Evaluation
Explore essential open-source benchmarks for evaluating database performance.
154. What Happened to Hadoop? What Should You Do Now?
by Monte Zweben & Syed Mahmood of Splice Machine
155. Solving Data Integration: The Pros and Cons of Open Source and Commercial Software
There was an awesome debate on DBT’s Slack last week discussing mainly two things:
156. 5 Reasons to Invest in Analytics For Your Startup Now
Data analytics are a startup's best friend, and here are five reasons why.
157. Could Blockchain and Big Data Come Together To Open Up A New Chapter in Data Integrity?
Whenever the term “Blockchain” comes across, many relate it with cryptocurrencies like Bitcoin. Yes, this technology has truly transformed the world of virtual currencies by speeding up transactions, providing privacy and transparency, and many more.
158. How to Improve Data Quality in 2022
Poor quality data could bring everything you built down. Ensuring data quality is a challenging but necessary task. 100% may be too ambitious, but here's what y
159. ELT is Dead, and EtLT Will End Modern Data Processing Architecture
Why EtLT is gradually replacing ETL and ELT as the global mainstream data processing architecture?
160. Distributed Storage is the Best Data Storage Tool for The Metaverse
The most suitable data storage tool for Metaverse is undoubtedly distributed storage.
161. Data Privacy is Becoming More Important for Users in 2022
A look at how data privacy is becoming more important for users in 2022
162. How Synthetic Data is Accelerating Computer Vision
In the spring of 1993, a Harvard statistics professor named Donald Rubin sat down to write a paper. Rubin’s paper would go on to change the way that artificial intelligence is researched and practiced, but its stated goal was more modest: analyze data from the 1990 U.S. census, while preserving the anonymity of its respondents.
163. Big Data Analysis for the Clueless and the Curious
Big data analytics has been a hot topic for quite some time now. But what exactly is it? Find out here.
164. A High Level Explanation of Data Types for Decision Makers
There are three different types of data: structured data, semi structured data, and unstructured data.
165. The Failed Promises of Extract, Transform, and Load—and What Comes Next
Faster, Better Insights: Why Networked Data Platforms Matter for Telecommunications Companies
166. A Quick Guide To Business Data Analytics
For many businesses the lack of data isn’t an issue. Actually, it’s the contrary, there’s usually too much data accessible to make an obvious decision. With that much data to sort, you need additional information from your data.
167. Essential Databases Every Developer Should Be Familiar With
Here's that every kind of database that every developer to know about.
168. 20 Herramientas de Inteligencia Empresarial (BI) más Populares en 2020
Business Intelligence (BI) es un negocio basado en datos, un proceso de toma de decisiones basado en datos recopilados. A menudo es utilizado por gerentes y ejecutivos para generar ideas procesables. Como resultado, BI siempre se conoce indistintamente como "Business Analytics" o "Data Analytics".
169. How Big Data and Artificial Intelligence Will Go Hand in Hand?
The emergence of technology is playing an inevitable role in business. It’s drastically transforming the way people work together in an organization. Both these technologies are revolutionizing every aspect of our life. These technologies are creating a culture where the collaboration of IT leaders and businesses results in realizing values from all generated data.
170. Aerospike Graph: the Latest Entry in the Graph Database Market
The story behind the birth of a new entry in the graph database market and its differentiation in a very densely populated market.
171. Best Types of Data Visualization
Learning about best data visualisation tools may be the first step in utilising data analytics to your advantage and the benefit of your company
172. Powering the Future: Decentralized Oracles and Metaverse DNA
In the decade-long history of blockchain and distributed ledger technology (DLT), rapid developments have led to consistent advances in the capabilities of decentralized financial platforms. By today’s standards Bitcoin has its limits: it supports value transfer and the storage of metadata within those transfers, but little else. With a block time of 10 minutes and a maximum block size of roughly four megabytes, it is also extremely slow compared to the emergent blockchains of the past few years.
173. How High-Quality Datasets Can Revolutionize Business Outcomes with Machine Learning
The accuracy of a machine learning model is a measure of how well it can make predictions on new, unseen data.
174. How Can We Harness Technology to Help Prevent Mass Shootings?
Mass shootings are tragic but increasingly common — could technology help curb the violence?
175. How to Create Bullseye Charts with JS: COVID-19 Vaccine Pipeline
Bullseye charts are widely used in drug pipeline & clinical trials data analysis. Learn how to create one in JavaScript and explore the COVID vaccines by phase.
176. Not Only Python: Problems, Errors and Alternatives
In this article, we will explore the emergence of new machine learning languages, how they have eroded Python's market share.
177. Getting Started with Data Visualization: Building a JavaScript Scatter Plot Module
Scatter plots are a great way to visualize data. Data is represented as points on a Cartesian plane where the x and y coordinate of each point represents a variable. These charts let you investigate the relationship between two variables, detect outliers in the data set as well as detect trends. They are one of the most commonly used data visualization techniques and are a must have for your data visualization arsenal!
178. Advancing Observability Platforms: Upgrading Data Processing and Reducing Costs with Apache Doris
Discover how GuanceDB elevates observability with Apache Doris, slashing costs by 70% and boosting data query performance by 200-400%.
179. Common RAID Failure Scenarios And How to Deal with Them
Most businesses these days use RAID systems to gain improved performance and security. Redundant Array of Independent Disks (RAID) systems are a configuration of multiple disk drives that can improve storage and computing capabilities. This system comprises multiple hard disks that are connected to a single logical unit to provide more functions. As one single operating system, RAID architecture (RAID level 0, 1, 5, 6, etc.) distributes data over all disks.
180. A Guide on The Future of ETL: EL(T) not ELT
How we store and manage data has completely changed over the last decade. We moved from an ETL world to an ELT world, with companies like Fivetran pushing the trend. However, we don’t think it is going to stop there; ELT is a transition in our mind towards EL(T) (with EL decoupled from T). And to understand this, we need to discern the underlying reasons for this trend, as they might show what’s in store for the future.
181. Data-driven Marketing: Unleashing the Power of Big Data for Targeted Campaigns
In today's digital era, the abundance of data has transformed the way businesses approach marketing.
182. What is RFM (Recency, Frequency, Monetary) Analysis?
RFM analysis is a data-driven customer segmentation technique that allows marketing professionals to take tactical decisions based on severe data refining
183. Why We Should Have Different Databases
Today there are hundreds of SQL and NoSQL databases. Some of them are popular, some are ignored. Some are user-friendly and well documented and some are hard to use. Some are open sourced and some are proprietary. And, perhaps, the most important - some are scalable, optimized, highly available and some are difficult to scale or maintain.
184. Data Drama: Navigating the Spark-Flink Dilemma
Explore Apache Flink and Spark in real-world business scenarios. Choose the right tool for your big data needs
185. From Centralized to Federated: Evolving Data Governance Operating Model
See how a federated data governance model address challenges of centralized systems by enabling flexibility, regulatory compliance, and innovation for business
186. How The 5th Wave Of Computing And IoT Are Changing Our Lives
According to the World Economic Forum, in 2020 the entire digital universe has reached 44 Zetabytes of Data.
187. Database APIs vs Datasets: Weighing Benefits, Drawbacks, and Transition Strategies
Database API is a convenient way to get relevant data records whenever needed. Learn about the benefits, limitations, and common use cases.
188. The Essential Data Cleansing Checklist
After some time working as a data scientist in my startup, I came to a point where I needed to ask for external help with your project.
189. Building a Data Management Strategy: Importance, Principles, Roadmap
Already routinely called the currency, the lifeblood, and the new oil of the modern business world, data promises organizations unbeatable competitive advantages.
190. Debunking The Top Myths Surrounding AI
Myths about artificial intelligence range from fearful reports of robots to outlandish expectations of the technology. Today, consumers encounter artificial intelligence continuously through smartphones, customer service centers, websites, and appliances. Surveys show that nearly nine in 10 Americans use some form of artificial intelligence device, and 79% of people report AI having a perceived positive impact on their lives. Despite the overwhelmingly positive uptake of the technology, films, art, and literature have long warned about the potential dangers of AI in science fiction storytelling. So, how much of this is based on reality?
191. How to Setup Your Organisation's Data Team for Success
Best practices for building a data team at a hypergrowth startup, from hiring your first data engineer to IPO.
192. 5 Big Data Trends for the Post-Pandemic Future
As the digital landscape continues to expand at a mind-boggling pace, the amount of data stored and used by enterprises also increases. Over the course of recent years, the accumulation of big data within organizations has slowly but surely, established itself as a staple within companies, particularly as far as generating data-driven insights and upholding security.
193. Using Arrow Flight SQL Protocol in Apache Doris 2.1 For Super Fast Data Transfer
Apache Doris 2.1 just got a major speed boost with Arrow Flight SQL for up to 10x faster data transfers.
194. Automate Submissions for the Numerai Tournament Using Azure Functions and Python
Python Automation with Azure Functions, to compete in the weekly Numerai tournament.
195. Why Data Governance is Vital for Data Management
Both data governance and data management workflows are critical to ensuring the security and control of an organization’s most valuable asset-data.
196. Zipping up Lambda Architecture for Faster Performance
Lambda segregates real-time and offline big data processing. Our pipeline implements separate pipelines for each data type, allowing for efficient processing.
197. How To Use Change Data Capture for Fraud Detection
Still relying on overnight processes to drive your decision making? Maybe it’s time to consider an evaluation of your CDC pattern that uses new technology.
198. The Critical Role of Customer Data in Creating Personalized Product Experiences
With 97.2% of businesses investing in data and AI, one thing is clear: data isn’t a “nice to have” anymore; it’s a necessity.
199. Accelerating Excavation and Refinement of Data Gold Mines
Unlock the potential of data-driven decision-making with generative AI and NLP.
200. Why "Big Data" is No Longer Relevant in the Age of Machine Learning and Deep Learning
Discover why "Big Data" is no longer relevant with the rise of Machine Learning and Deep Learning. Learn how these technologies transform data analytics.
201. How Big Data Reshapes the Future of Digital Advertising - 3 Examples
These days, big data is truly omnipresent. According to revenue forecasts, by 2026, big data volumes are expected to reach a whopping $92 billion. What August 2019 CMO Survey goes on to say is that the majority of ad tech and martech leaders agree - big data and innovative technologies are two pillars on which their marketing strategies are based. Businesses use big data in order to develop a detailed portrait of each segment of their customer base and apply these marketing strategies properly.
202. Why Microservices Suck At Machine Learning...and What You Can Do About It
203. Public Web Data for Business: Common Challenges And How to Solve Them
Businesses working with public web data experience various challenges. This article covers the most common ones and how to overcome them.
204. Holy Land of Crypto Users: How does Web3.0 Data Empower Centralized Exchanges?
Designing a data-oriented, user-incentive mechanism is a good path when developing the future of centralised exchanges for the cryptocurrency industry.
205. 361 Stories To Learn About Big Data
Learn everything you need to know about Big Data via these 361 free HackerNoon stories.
206. How Different Analyst Types Can Positively Impact Your Small Business
Data analysis used to be considered a luxury of big business.
207. Public Health Improvements as a Result of Data Usage and Analysis in Healthcare
Big data has made a slow transition from being a vague boogie man to being a force of profound and meaningful change. Though it’s far from reaching its full potential, data is already having an enormous impact onhealthcare outcomes across the world — both at the public and individual levels.
208. How Will Blockchain Fix the Centralization of Data?
“In order to have a standard of value [cryptocurrency] must stand outside all value schemes. It must have value in and of itself."
209. Beyond Data: The Rising Need for AI Security
As organizations increasingly deploy AI systems for decision-making, ensuring both data and AI pipeline security becomes critical to safeguard integrity, trust.
210. Why Self-Service Analytics Tools Are Important For Business Decisions Making
How to use Big Data, Self-Service Analytics Tools and Artificial Intelligence to Empower your Company Business Decisions Makers with State Of The Art Software
211. Writing Your Own Product Recommender? You Need To Read This First
212. Digital Technologies And Their Increased Role - What Does The Future Hold?
Digital technologies offer more and more new opportunities. The advancement of technologies makes our life easier and our planet a better place to live.
213. 8 Ways to Gather and Leverage Customer Data of Your Ecommerce Website
In this article, you will take a look at some of the different approaches you can use to gather and leverage customer data for your eCommerce website.
214. Building AI Products with Big Data
Credits: Thanks to our sponsor Amazon, the Advancing Women in Product Team: Keshav Attrey, Reeba Monachan Attrey, Kanika Kapoor, Alok Gupta, Jackie Yen, our AWIP volunteers and our panelists.
215. How Programming, AI, and Big Data is Giving Google A Chance to Save the World
Big business and saving the planet often do not go hand in hand, however in some cases they do. Take a look at how Google plans on saving the future with tech.
216. This New Data Type Is 8 Times Faster Than JSON: Improve Your Semi-Structured Data Analysis
Apache Doris provides a new data type: Variant, for semi-structured data analysis, which enables 8 times faster query performance than JSON with 1/3 storage.
217. Why AI Unified Analytics is Good for Your Business
AI unified analytics can help businesses collect and analyze the data that AI tools require. Learn more about how AI unified analytics is good for business!
218. Tamper Proofing in the Digital Age: A Look at Proof of SQL
Interview with Jay White discussing the ZK-Proof Proof and it's development.
219. The Added Value of GPU-Accelerated Analytics
GPUs are now being put to the test in the three fastest developing applications in today’s tech ecosystem.
220. High-Utility DeFi Data Analytics Tools For Crypto Investors
These four growing platforms will give investors the tools they need to make smarter decisions
221. The Future of Human In The Loop
Since the 1980’s, human/machine interactions, and human-in-the-loop (HTL) scenarios in particular, have been systematically studied. It was often predicted that with an increase in automation, less human-machine interaction would be needed over time. Human input is still relied upon for most common forms of AI/ML training, and often even more human insight is required than ever before.
222. How to Migrate Data from an MSSQL Server to PostGreSQL?
Thinking of shifting to a new database management engine? Here's how to migrate data from SQL server to PostgreSQL.
223. AML Compliance in 2025: How Financial Institutions Can Stay Ahead of Evolving Financial Crime Risks
Explore AML compliance trends in 2025 and banks stay ahead of evolving financial crime risks up.
224. Processing Massive Amounts of On Demand Data Without Crashing NodeJS Main Thread
Processing Massive Data On Demand Without Crashing NodeJS Main Thread
225. Computing on the Edge: How GPUs are Shaping the Future
Discover how GPU acceleration is reshaping data processing, offering unparalleled speed and efficiency for AI and big data analytics.
226. 4 Data Transformations Made Spreadsheet-Easy
Gigasheet combines the ease of a spreadsheet, the power of a database, and the scale of the cloud.
227. Big Data Analysis on Blockchain with CEO of Covalent, Ganesh Swami
I sat down with Ganesh Swami, co-founder and CEO at Covalent, a Blockchain Big Data analytics firm, to discuss the Ethereum ecosystem.
228. Understanding the Main Differences between Structured and Unstructured Data
In this, I explore structured, unstructured, and semi-structured data, as well as how to convert unstructured data, and AI’s impact on data management.
229. 6 Places to Start a Career in Data Science
How to become a data scientist?
Want to become a Data Scientist? Here are the resources.
Resources to Become a Data Scientist
230. 5 Industries That Rock Big Data Analytics
Each day we produce 2.5 EB of data [3]. This is 2.5 billion gigabytes of information about everything. This creates unlimited opportunities for collecting, processing, and analyzing vast amounts of both structured and unstructured data, also known as Big Data.
231. Do Database Administrators Still Matter in the Age of Managed Databases?
Discover how the database administrator role is evolving with managed services like AWS RDS.
232. Data Product Managers and the Data Mesh
With data becoming very ubiquitous in the enterprise, proper definition of a data product, its lifecycle and development process should be established.
233. What Are The Challenges of Monetizing and Selling Data?
There have been great advancements in monetization opportunities in the last decade, but there are still challenges when it comes to generating big data analyti
234. BitsCrunch Raises $3.6 Million from Coinbase Ventures, Crypto.com Capital and Animoca Brands
BitCrunch has raised $3.6 million in a private round of funding led by Animoca Brands, including Coinbase Ventures, Crypto.com Capital and Polygon Studios.
235. Using Artificial Intelligence and Big Data To Deliver Your Pitch
A good pitch tells the story of your idea. From its inception to its present form and everything in between. Utilising multimedia, graphs and visuals is a good way to keep your audience engaged and up to speed. Most fundamentally, using data is important for both your audience and your idea.
236. Universal Data Tool: New Skeletal/Pose/Landmark Annotation, Dutch, and Convert Options
For those who haven’t heard of the Universal Data Tool, it is an open-source web or desktop program to collaborate, build and edit text, image, video, and audio datasets with labels and annotations.
237. Data Scientists, Software Engineers And The Future of Medicine
The world is changing, especially the way we cure ourselves. The rise of next generation computing, cloud computing technologies, AI, decentralization, etc. have dramatically changed seemingly every industry. Computational Medicine is now an emerging new discipline.
238. Machine Learning Trends Businesses Should Know In 2020
Have you ever considered how much data exists in our world? Data growth has been immense since the creation of the Internet and has only accelerated in the last two decades. Today the Internet hosts an estimated 2 billion websites for 4.2 billion active users.
239. Streaming Wars: Why Apache Flink Could Outshine Spark
Comparing Apache Flink & Apache Spark in stream data processing. Exploring architectural nuances, applications, and key distinctions between the platforms.
240. How to Improve Query Speed to Make the Most out of Your Data
In this article, I will talk about how I improved overall data processing efficiency by optimizing the choice and usage of data warehouses.
241. Allstate's Car Insurance Algorithm: How Insurance Algorithm Was Analyzed
State regulators and consumer advocacy groups have scrutinized Allstate Corporation’s use of big data and personalized pricing in the way it calculates how much the company charges its private auto insurance customers.
242. The Ways in Which Big Data can Transform Talent Management and Human Resources
Big Data is changing human resource management for good. We explore 4 major ways data analytics is upending & expanding the role of human resource departments.
243. Top 7 Trends of Digital Transformation in Higher Education
Higher Education is highly influenced by today's digital transformation and technological advances. The student learning experience can be boosted with the use
244. Neo4j Is Building an Ecosystem of Graph-powered Features for Generative AI
Graph database Neo4j is building an ecosystem for Graph-powered features for Generative AI and beyond with all major cloud platforms
245. How Big Data is Shaping Adaptive Learning
Big data analytics will likely drive more widespread adoption of adaptive learning tools, especially regarding big data for education and learning environments.
246. How Important is the API Economy for Blockchain Application Development?
247. AWS Snow Family: An Old Solution to a New Problem
The AWS Snow Family is a group of three products that solved the problem of slow data transfers and edge computing associated with cloud storage.
248. Compression in Big Data: Types and Techniques
This article will discuss compression in the Big Data context, covering the types and methods of compression
249. The Noonification: Understanding How Data Warehousing on AWS Works (12/3/2023)
12/3/2023: Top 5 stories on the Hackernoon homepage!
250. Building Sustainable AI/ML Solutions in the Cloud with Federated Learning
Compared to centralized training and cooling mechanisms adopted at data centers, how can Federated Learning help us combat detrimental environmental impacts?
251. How to Achieve Optimal ROI Through Process Mining
Explore the evolution of process mining since 2011, focusing on advancements by industry leaders like Celonis and UiPath.
252. How to Analyze and Process Unstructured Data in 5 Simple Steps
In this article, we’ll look at how to analyze and process unstructured data while using business intelligence tools to simplify the entire process.
253. How to Back Up Exchange Online Data
254. Turn Big Data into a Big Success: 5 Tips for Effective Big Data Analytics
Organizations must acquire appropriate measures for turning their big data into a big success.
255. Lambda Architecture: A Comprehensive Introduction and Breakdown
Big data is on the rise, and data systems are tasked with handling it. But this begs the question: Are these systems up for the task?
256. Supporting 'Citizen IT': It’s Critical to Democratize Your Data
Democratizing data to enable Citizen IT provides a competitive advantage to organizations - here's why.
257. AI Just Took Over Ad Targeting—And It’s Smarter, Faster, and Less Creepy Than Ever
Next-gen AI ad platforms use vector databases, indexing, and privacy-aware AI for real-time optimization, boosting ad spend efficiency while staying compliant.
258. 10 Most Evolving Big Data Technologies to Catch Up on in 2022
At the heart of it all, big data also has a dark side. Several tech giants are facing heat from the public and government regarding the issue of data privacy.
259. 5 Essential Product Classification Papers for Data Scientists
Product categorization/product classification is the organization of products into their respective departments or categories. As well, a large part of the process is the design of the product taxonomy as a whole.
260. Effective Use of Big Data and Analytics for Business Ventures
Business data analytics is often a very complex and intensive process to execute. In the era of big data analytics where a large set of varied data needs to be analyzed in order to uncover insightful information, things become more complex. However, such a comprehensive data
analysis model will help uncover various hidden patterns, market shifts, and trends, unknown correlations, customer behavior, etc. Getting an actionable insight into these will help the organizational decision-makers to make well-informed decisions.
261. Data Privacy: Why The Existing Architecture is in Dire Need of Evolution
Data is to the 21st century what oil was for the 20th century. The importance of data in the 21st century is conspicuous. Data is behind the exponential growth witnessed in the digital age. Increased access to data, through the internet and other technologies, has made the world a global village.
262. Database in Fintech: How to Support 10,000 Dashboards Without Creating a Mess
As your business grows, your data management will arrive at a point when "standardization" is needed.
263. Blockchain and AI | A Disruptive Alliance
AI and Blockchain are among some of the most influential drivers of innovation today — a natural convergence is occurring.
264. The Collective Loves Data: How Big Data Is Shaping and Predicting Our Future
Big data shapes our future! Explore how massive datasets are used to predict trends & make smarter decisions.
265. Machine Learning Explained in 5 Minutes
Google uses it to provide millions of search results every hour. It helps Facebook guess your next love interest. Even Elon Musk’s Tesla uses it to make self-dr
266. ACID Transactions: Fundamentals of Delta Lake - Part 1
Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads.
267. 253 Stories To Learn About Data Analysis
Learn everything you need to know about Data Analysis via these 253 free HackerNoon stories.
268. Analyzing Montreal’s BIXI Ridership With Data And Visuals
Been to Montreal? Have you heard of the term bixi? Well, this article will educate you about bixi ridership and the factors that affect it.
269. 229 Stories To Learn About Data Analytics
Learn everything you need to know about Data Analytics via these 229 free HackerNoon stories.
270. 5 Ways to Become a Leader That Data Engineers Will Love
How to become a better data leader that the data engineers love?
271. 8 Reasons Why Inventors Should Try This Free and Open-Source Patent Search Engine
PQAI is a free and open-source patent search engine that uses artificial intelligence to search for patents using queries in natural language.
272. A Brief Introduction to 5 Predictive Models in Data Science
Predictive Modeling in Data Science is more like the answer to the question “What is going to happen in the future, based on known past behaviors?”
273. Visualizing IoT Data with MQTT, QuestDB, and Grafana
Time-series data is crucial for IoT device monitoring and data visualization in industries.
274. How to Analyze and Visualize the Game of Thrones Character Relationships
The hit series Game of Thrones by HBO is popular all over the world. Besides the unexpected plot twists and turns, the series is also known for its complex and highly intertwined character relationships. In this post, we will access the open source graph database Nebula Graph with NetworkX and visualize the complex character connections in Game of Thrones with Gephi.
275. SubQuery to Provide Indexing and Querying Infrastructure to Developers on Algorand
SubQuery is a blockchain developer toolkit that makes it easier to build upcoming Web3 apps.
276. Web3.0 Powered Privacy: Decentralization for More Control and Transparency
A look at the importance of data privacy in today's digital age, where personal information is being collected, used, and shared at an unprecedented rate.
277. 5 Big Data Problems and How to Solve Them
“Big Data has arrived, but big insights have not.” ―Tim Harford, an English columnist and economist
278. Your Credit Card Has Become a Data-Mining Machine
Discover how card payments work: types of cards, key players, open vs. closed loop networks, and the role of AI/ML and Big Data.
279. Workload Isolation in Apache Doris: Optimizing Resource Management and Performance
Apache Doris supports workload isolation based on Resource Tag and Workload Group and provides solutions for different tradeoffs.
280. How to Use Big Data and Artificial Intelligence for Demand-Based Pricing in Retail
You can call yourself a guru of retail pricing if you can make the right pricing decisions for every one of your products, separately and combined, based on their demand elasticity at any given moment.
281. Kannada-MNIST:A new handwritten digits dataset in ML town
TLDR:
282. Comparative Study Of Best Time-Series Models For Pandemic Response
With the effect of the pandemic increasing every day and casting a vehemently toxic influence in almost all parts of the world, it becomes important how can we contain the spread of the disease. In an effort to combat the disease every country has increased not only their testing facility but also the amount of medical help and emergency and quarantine centers. Here in this blog, we try to model Single-Step Time Series Prediction, using Deep Learning Models, on the basis of Medical Information available for different states of India.
283. Facebook: The Magic 8 Ball
It is easier for a camel to pass through the eye of a needle than for a homo sapien to quit this junk.
284. What is Big Data in Healthcare and How is it Used?
The pandemic is having an enormous impact on the healthcare sector. Between overwhelming hospitalization rates, intensifying cybersecurity threats, and an aggravating number of mental illnesses due to strict lockdown measures, hospitals are desperately searching for help. Big data in healthcare seems like a viable solution. It can proactively provide meaningful, up-to-date information enabling clinics to address pressing issues and prepare for what’s coming.Hospitals are increasingly turning to big data development service providers to make sense of their operational data. According to Healthcare Weekly, the global big data market in the healthcare industry is expected to reach $34.3 billion by 2022, growing at a CAGR of 22.1%.So, what is the role of big data analytics in healthcare? Which challenges to expect? And how to set yourself up for success?
285. How to Get Qualified to Work in Big Data for Decision Intelligence
Decision intelligence, Data Stories, and Data Cloud Services are the three trends that are ranking high in the Data Analytics 2021.
286. Data Integrity: What is It, and Why Does It Matter?
Discover why data integrity is critical for AI, Big Data, and decision-making. Explore how ZK technology and Horizen 2.0 offers secure solutions
287. Five Ways to Cut Infrastructure costs in High-load Systems: AdTech Case Study
5 ways to cut infrastructure costs in high-load systems: AdTech case study
288. Big ‘Earth Observation’ Data: Challenges and Applications
As nearly a thousand Earth observation satellites currently orbit the planet, terabytes of remote sensing data and satellite imagery of land, vegetation, water bodies, glaciers, urban landscapes, and other geographic features become available for end users across multiple industries. Modern GIS systems allow the collection of all such geospatial data in one place for a comprehensive analysis of the area under study.
289. Data Analytics is a Journey
It is 2020 and the data analytics has gained so much attention even outside of the tech community. "Data is gold", they say - no one wants to be left behind. However, getting the right strategy is neither a straightforward nor static process.
290. Unlocking the Power of Advanced Data Types in Big Data
Features of the specialized data types near integers and strings, which we use in every-day life, will allow us to store and operate complex data structures.
291. Interpretation of Visualizations of Soil Data and Weather APIs
Learn how to visualize and interpret weather APIs and soil data in different graphs using python libraries, and Google Collab.
292. How Different Industries Put Data Analytics to Use
You must have heard about big data and the theory used behind it. However, are you aware of the top industries where data analytics is being used for changing the way we work in the actual world? Let's take a close look at the top big data industries and how they are getting reshaped by using data analytics. The main idea behind using big data is that it is a new method for gaining insight into the challenges faced by various companies each day. In earlier days it was not possible to collect and interpret a vast quantity of data because there was no technology available.
293. Distributed Data Store and Transaction Sagas
This is a tutorial on how to create a distributed data store by implementing a leader based replication.
294. 5 Simple Ways to Kickstart Your Freelance Data Science Career
If you’ve been itching to get your feet wet in the field, these steps will provide you with lots of valuable ideas and suggestions to kickstart your career.
295. Leveraging Big Data for Startup Growth in 2025
Discover how startups can leverage big data for growth in 2025. Learn strategies for data-driven decision-making, customer insights, and operational efficiency.
296. Predictive Data Mining Can Help Forecast the Online Behavior of Consumers (Podcast)
In this episode, we discuss how the company first began, how it has grown, and the solutions it currently offers.
297. Data Quality: Its Definitions And How to Improve It
Utilizing quality data is essential for business operations. This article explores data quality definitions and how to maintain it for everyday use.
298. Deepfakes: Thy Expiration Date is Nigh
Predictions that deepfake videos will keep getting better are not matched by the realities of the technology. Here's a sober look at the problems.
299. What Is A Data Mesh — And Is It Right For Me?
Ask anyone in the data industry what’s hot and chances are “data mesh” will rise to the top of the list. But what is a data mesh and is it right for you?
300. 4 Ways Data Science Helps Streamline Business Operations
Data Science has changed the way organizations collect, analyze, and process different types of information.
301. How Machine Learning is Transforming Biotech
Machine learning is re-writing everything we thought we know about what's possible through biotech.
302. What Personal Details Are You Sharing Without Knowing?
Unless you have changed your web browser default settings it is quite likely you are leaking personal details as you move around online. But just how much?
303. The People and Tech Behind Data Science
What is a data scientist? The job has been around for hundreds of years, though as you may suspect things have changed significantly, especially over the last century. In the 1740s Bayes’ Theorem posited that when new data was added to an existing belief, the result was a new and improved belief. This is the basis for the scientific method, by which scientists discover better and better explanations for things. When applied to data, the scientific method creates data science, in which data scientists can use the piles of data people are generating to discover new and better predictions about the future.
304. Machine Learning, 5G and Data Science Will be Critical to the Future of the Internet of Things
By 2020, the total number of Internet-connected devices will be between 25-50 billion.
305. Data Lineage is Like Untangling a Ball of Yarn
Data lineage is a technology that retraces the relationships between data assets. 'Data lineage is like a family tree but for data'
306. Blockchain Protects From Data Miners But Is Also A Perfect Tool For Data Mining
The article tells what happens when blockchain meets online advertising.
307. Auto-Synchronization of an Entire MySQL Database for Data Analysis
Flink-Doris-Connector 1.4.0 allows users to ingest a whole database containing thousands of tables into Apache Doris, a real-time analytic database, in one step
308. Machine Learning for Fraud Prevention
309. It’s in the Data: How COVID-19 is Affecting the Digital Landscape
310. Discover Funnel Bottlenecks: Step-by-Step Analysis with BigQuery
Learn how to use BigQuery for e-commerce funnel analysis. Track user transitions between steps like “add to cart” and “purchase,” and identify where to improve
311. The Hidden Tax of Cloud BI: Zombie Data Movement Between Platforms
Hidden cloud BI cost: data egress between platforms. Learn how “zombie data movement” quietly inflates analytics bills in modern BI architectures.
312. Is Data Monetization Dead?
The advent of cryptocurrency and web3 has led to investigations and experiments into what ways could a total decentralized digital society manifest.
313. Web Scraping for Good: Utilising the Power of Data Ethically
How can web scraping deliver a significant positive impact and serve non-profit, socially important causes?
314. Legal Billing Software for Busy Lawyers, an Overview
Lawyers, accountants and auditors who are typically paid by the hour, have a hard time getting paid what they are truly owed. Legal billing software may just help.
315. 96 Stories To Learn About Data Engineering
Learn everything you need to know about Data Engineering via these 96 free HackerNoon stories.
316. Data in AI: A Deep Dive With Jerome Pasquero
How is Data Transforming AI - The What's AI Podcast (episode 27)
317. Small Businesses use AI Tools to Increase Their Leads By 50%
AI can empower sales reps by monitoring different signals and predicting a specific lead's readiness to purchase. AI tools can reduce customer acquisition costs
318. Getting to Know Google Analytics 4: Four Smart Features You Don’t Know About
Let’s take a deeper look into Google Analytics 4 and explore some of its key features that you might not yet know about.
319. A Brief Introduction to Recommendation Systems
Recommendation systems offer relevant product suggestions to users by using machine learning based on data gathered. More so,it uses characteristics information
320. A Hands-On Guide to Inverted Indexes: Accelerate Text Searches by 40
This post is a deep dive into the inverted index and NGram BloomFilter index, providing a hands-on guide to applying them for various queries.
321. Declarative Engineering: Using Terraform to Code Your Data Pipelines
A small modern data stack that ETLs data from a PostgreSQL database into a ClickHouse database.
322. Build your Dataset from COCO with the Universal Data Tool
If you haven’t heard of the Universal Data Tool yet, it’s an open-source web or desktop program to collaborate, build and edit text, image, video, and audio datasets with labels and annotations.
323. Training AI to Gauge Online Reputation and Make the Market Safer
When looking for a trustworthy business partner, quality service in your neighborhood, or hiring new talent, you can rely on AI to measure reputations online.
324. How to Use Public Keys in Data Lifecycles
The data lifecycle (also known as the information lifecycle) refers to the full-time period during which data is present in the system.
325. Apache Airflow And Its Contribution to Enterprise Data Integration
Large enterprises, which have more than one business unit, usually have more than one data platform environment.
326. Understanding Kafka with Factorio
[https://hackernoon.com/photos/zXE6n93W9qXDNsP8hiip4vQhbqg2-pi1dk2f60]
This is a 1.0 story that I edited
Thanks to Tom de Ruijter, [https://medium.co
327. How to Build a Scalable Data Mesh in AWS with Lake Formation
Learn how to implement a secure, scalable AWS Data Mesh using Lake Formation, Glue, Lambda, and IAM for cross-account data sharing and governance.
328. How AI-Powered Data Mapping is Democratizing Data Management
Learn how AI-powered data mapping is transforming data management, making it more accessible and efficient for everyone.
329. Listen to That Poor BI Engineer: We Need Fast Joins
Yes, you can expect fast joins from a relational database.
330. Mastering Real-Time Data: Rahul Chaturvedi's Strategies for Building Reliable Data Platforms
Rahul Chaturvedi is a Staff Software Engineer at Uber Technologies Inc. He has been at the forefront of optimizing one of the world's largest Kafka deployments.
331. As AI Gets More Emotionally Intelligent, So Must We
How people behave in solitude is vastly different than how they behave in public, but the foundation of one’s persona remains constant. Dancing around the apartment when nobody’s watching expresses a secret desire to do so on a grand stage, but humans modulate those whims as societal norms dictate.
332. Fixing Garbled Text When Syncing Oracle to Doris with SeaTunnel 2.3.9
When using SeaTunnel 2.3.9 to sync data from Oracle to Doris, you may encounter garbled characters.
333. From Raw Data to Actionable Insights: The Power of Data Aggregation
This article examines data aggregation processes: collecting data to present it in summary form.
334. How to Use Public Web Data for Talent Intelligence and Sourcing
Learn how public web data can boost your talent sourcing efforts in both quality and quantity.
335. Thrilled to be Recognized as Contributor of the Year - Data Science & Data Analytics
Hooray! We have made it to the Hackernoon Awards. Xtract.io, the data provider's company is happy and elated to be part of #noonies2021. Join us in our victory!
336. Data Lakes Are Crucial to Business Analytics and Big Data Processing
Big data is a sort of Data addition that contains greater variety, arriving in increasing volumes and with more velocity which is also called three Vs. It could explain in several words by severals but actually what stands for it.
337. What Are the Key Differences Between Qualitative and Quantitative Data?
This article uncovers the key differences between qualitative and quantitative data with examples.
338. Feature Stores 2.0: The Next Frontier of Scalable Data Engineering for AI
Revolutionizing AI pipelines with scalable, real-time Feature Store innovation.
339. Financial Anti-Fraud Solutions Available on the Apache Doris Data Warehouse
This post will get into details about how a retail bank builds their fraud risk management platform based on Apache Doris and how it performs.
340. Big Data's Influence on Decision Making in the Healthcare Industry
Big data is transforming decision-making in healthcare and this article explores how it can be used to improve patient care, as well as its challenges.
341. Java or Python: Which One Should a Data Scientist Learn?
Data science is one of the most promising fields in tech. To succeed in the field, mastery over programming languages like Java and Python is essential.
342. 3 Industries Harnessing the Power of Big Data: Healthcare, Law, and Retail
Big Data's value, popularity, and scale of usage in business today come from a few of the indisputable benefits it has to offer:
343. Python vs. Spark: When Does It Make Sense to Scale Up?
Wondering when to switch from Python to Spark? This practical guide breaks down the real differences, warning signs, and best use cases—so you know exactly when
344. Apache DolphinScheduler Brings Major Enhancements and Performance Upgrades in Latest Release
Apache DolphinScheduler 3.3.0-alpha is here!
Featuring massive improvements: DSIP architecture upgrades, remote/audit logs, new plugins.
345. Apache Arrow: Optimizing PySpark Applications
Apache Arrow eliminates PySpark serialization bottlenecks. Learn how columnar, zero copy memory boosts Pandas, Spark, and UDF performance at scale.
346. Paying Crypto Taxes: Nuisance or Cost of Doing Business?
TRASTRA founder and CEO Roman Potemkin on what is right, wrong, and unclear with implementing crypto taxes.
347. Essential 2024 AI & Big Data Conferences: Oxycon, World AI Summit, AI Summit NY, Analytics Summit
World Summit AI, AI Summit New York, Oxycon, Analytics Summit, and Big Data Conference Europe lead the list of AI and Data conferences you should check out.
348. Introduction to Delight: Spark UI and Spark History Server
Delight is an open-source an cross-platform monitoring dashboard for Apache Spark with memory & CPU metrics complementing the Spark UI and Spark History Server.
349. Who is a Data Engineer and What Do They Do
As a data engineer, your job involves handling lots of information (we call it data).
350. 5 Skills Every Successful MLOps Engineer Should Have
Discover the five key skills every successful MLOps Engineer should have. Elevate your MLOps career with these crucial insights.
351. Go Clean to Be Lean: Data Optimization for Improved Business Efficiency
The article discusses cost optimization with clean data, explaining how businesses can save resources by reducing the workload for data analysts and more.
352. A Brief Introduction to Commit Logs
Logs are everywhere in software development. Without them there’d be no relational databases, git version control, or most analytics platforms.
353. Your Next Movie Watch Could Be From a Superapp
Content super apps could represent the next stage in the evolution of online streaming platforms.
354. Get Started With Big Data Analytics For Your Business.
Everything we do generates Data, therefore we are Data Agents. The question is: how we can benefit from this huge amount of data generated every day?.
355. 3 Things I Learned Building My First Neural Network
I’ve been working with massive data sets for several years at companies like Facebook to analyze and address operational challenges, from inventory to customer lifetime value. But I hadn’t worked yet on something this ambitious.
356. Machine Learning Concepts In Python For your Next App
Python can be used in machine learning, especially through using these basic machine learning concepts as building blocks for data analysis and other functions.
357. Using Upsolver To Get Insights Into Your Company's Big Data
Upsolver is a no-code data lake engineering platform for agile cloud analytics. Let's see how easy it is to use.
358. Automated Data Catalogs will Help Manage Data in 2022
Data is increasingly playing a dominant role in business. Know how automating your data catalog can help with efficient data management in 2022.
359. Navigating Architectural Trade-offs at Scale to Meet AI Goals in 2026
Success in 2026 is predicated on having total clarity of the underlying data infrastructure.
360. Data Analytics: Apache Doris' Impact in Reporting, Tagging, and Data Lake Operations
Delve into Apache Doris, a data powerhouse revolutionizing analytics for fintech with high-performance and scalable operations.
361. The Hitchhiker's Guide to pySpark DataFrames
Big Data has become synonymous with data engineering. But the line between Data Engineering and Data scientists is blurring day by day. At this point in time, I think that Big Data must be in the repertoire of all data scientists.
362. SQL Databases Vs. NOSQL Databases
The decision to choose a database for project is not that simple. But when it comes to choosing a database, the biggest decisions is picking a relational (SQL) or non-relational (NoSQL) data structure.
363. Emerging Food Technology Trends & Insights for 2022
The food industry is one industry that benefits from the use of technology, from data gathering, food quality, blockchain tech and supply chain tracking.
364. Introduction To Amazon SageMaker
Amazon AI/ML Stack
365. Efficient Data Management and Workflow Orchestration with Apache Doris Job Scheduler
Apache Doris 2.1.0's built-in Job Scheduler simplifies task automation with high efficiency, flexibility, and easy integration for seamless data management.
366. How to Improve VC Deal Sourcing Using Public Web Data
Learn how public web data can help you improve your deal sourcing methods.
367. How to Clean and Verify Address Data 'Without Using Code'
Today, data verification has become one of the greatest assets of an organization.
368. Ways To Overcome Linguistic Barriers with Language Technologies
COVID-19 has impacted every other industry and has made people adopt newer norms. The traditional translation industry is no different. Several disruptions have been introduced to keep things moving, thanks to Big data and machine translation technologies that have enabled the world to do business as usual.
369. What Kind of Skills Are Required to Become a Data Analyst?
Discover the essential skills required to become a successful data analyst, including technical tools, analytical abilities, and key competencies for thriving.
370. How to Turn Messy Healthcare Ops Data Into ML-Ready Features
Learn how to turn messy healthcare ops data into validated, explainable, and reproducible ML-ready features that hold up in production.
371. The Big Impact of Big Data on Businesses Today
The business impact companies are making with big data analytics is driving investment in digital transformation across the board.Faced with multiple waves of disruption in a COVID-19 world, almost 92% of companies are reporting plans to spend the same or more on data/AI projects, according to a recent survey from NewVantage Partners.Small wonder.Data mature companies are citing business-critical benefits from using big data, including:
372. How Big Data Can Help to Analyze Social Media Performance
During the last decade, social networking sites/apps have become the most important channels of communication.
373. Data-Driven Decisions at Scale: A/B Testing Best Practices for Engineering & Data Science Teams
Ship features like scientists: randomize, measure, and learn fast.
374. Top 5 Factors Behind Data Analytics Costs
A custom integrated data analytics solution would cost at least $150,000-200,000 to build and implement.
375. Data Potential: 10 Reasons Apache Iceberg and Dremio Should Be Part of Your Data Lakehouse Strategy
Discover the powerful synergy of Apache Iceberg and Dremio, revolutionizing data management and analytics.
376. Enterprise Blockchain for SmartCities
What is SmartCity?
377. 4 Ways in Which Predictive Analytics in Insurance is Paving the Way for the Future
Predictive analytics in insurance is radically changing the way companies do business. It will soon be at the core of countless new technology solutions.
378. Top Data Analyst Skills in 2021
Enhance your knowledge and skills in the field of data analytics with the help of data science certification for a rewarding career as a data analyst.
379. Allstate's Car Insurance Algorithm: How Insurance Algorithm Squeezes Big Spenders
Seven years ago, Allstate Corporation told Maryland regulators it was time to update its auto insurance rates. The insurer said its new, sophisticated risk analysis showed it was charging nearly all of its 93,000 Maryland customers outdated premiums. Some of the old rates were off by miles. One 36-year-old man from Prince George’s County, Md., who Allstate said in public records should have been paying $3,750 every six months, was instead being charged twice that, more than $7,500. Other customers were paying hundreds or thousands of dollars less than they should have been, based on Allstate’s new calculation of the risk that they would file a claim.
380. Mastering the Complexity of High-Volume Data Transmission in the Digital Age
Article explaining the importance of speedy data analytics and implementation of robust data infrastructure to achieve the same with live streaming data.
381. The Types and Benefits Of Cloud Computing
In this article, we discuss the options available for businesses to make the correct choice in terms of cloud computing to complement a business' needs.
382. The Gartner Hype Cycle Report and the Future of Data
Gartner identifies data labeling as one of the key factors responsible for the ongoing evolution of AI technology and rapid AI-powered product development.
383. Smarter Systems, Less Hassle: Inside DBMS Auto-Tuning
This paper provides a comprehensive analysis of automatic database management system (DBMS) tuning.
384. IoT, Big Data and the Era of the Zettabyte
Have you heard about the Internet of Things and Big Data? They are two very trending technologies that have evolved independently for a long time.
385. Big Data as the New Compass of Competition
Big Data Analytics has evolved into the modern organization’s most powerful compass.
386. Learn the Best Methods for Tuning DBMS Configurations
Explore the latest DBMS tuning techniques, including BO, RL, and optimization methods, for enhancing database performance.
387. The Economics of Web Data: ROI
A back-of-the-envelope way to estimate how much your client can spend on web data.
388. 89 Stories To Learn About Big Data Analytics
Learn everything you need to know about Big Data Analytics via these 89 free HackerNoon stories.
389. What Is Big Data? Understanding The Business Use of Big Data Analytics
Big data analytics can be applied for all and any business to boost their revenue and conversions and identify their common mistakes.
390. Mitigating Data Exfiltration: Four Ways to Detect and Respond to Unauthorized Data Transfers
Learn how to safeguard your data from unauthorized transfers with these 4 effective detection and response strategies.
391. Why Datasets are Crucial to Data Science: the Key to Informed Decisions
Datasets are crucial for anyone wanting to learn data science.
392. How Bayesian Optimization Speeds Up DBMS Tuning
Discover how Bayesian Optimization is revolutionizing DBMS configuration, making tuning faster and smarter for better performance.
393. Uncovering Data Debt : A Diagnostic Framework for Investigating Model Performance Degradation
ML models can quietly drift into irrelevance, causing hidden business impact. A strong diagnostic framework helps detect data debt and keep systems healthy.
394. Product manager dead after ‘taking a step back’ off cliff
395. How Big Data is Transforming Wealth as 28 Million UK Adults Use AI to Support Financial Decisions
The artificial intelligence boom has had a profound impact on fintech and the access of UK residents to more comprehensive financial services.
396. Fast-Coresets: A Nearly-Linear Time Algorithm for Efficient Clustering
Discover a nearly-linear time coreset algorithm for k-means and k-median clustering.
397. Investors Clamor for Digestible Data Analytics in the Fledgling Crypto Industry
As DeFi data generation grows with the industry, there is an increased need for platforms that are able to digest and analyze this data for investors.
398. Improving Healthcare Analytics and Implementation with Talend
Talend Open Studio for Data Integration can benefit direct marketers, offering several inbuilt Business Intelligence tools.
399. Processes and New Technologies in Data Transformation
In this article, I explore the benefits, types, and processes of data transformation and how it contributes to data management, integration, and new technologie
400. Scaling Off-Chain Data and Computation for Smart Contracts
As storing information on the blockchain becomes more popular, the availability of smart contracts becomes more widespread. They behave according to established parameters, automatically letting events happen once specified conditions are met.
401. The Benefits And Core Processes of Data Wrangling
This article examines the process and methods of data wrangling: preparing data for further analysis by transforming, cleaning, and organizing it.
402. Coresets, Compression, and the Quest for Faster Data Clustering
Explore fast and scalable clustering techniques for large datasets.
403. Why Modern BI Architectures Need More Than Just Star Schemas
Modern BI workloads demand more than star schemas. Learn when dimensional models work and when purpose-driven analytical tables improve performance.
404. A Dive into Education Tech Trends: Embracing Innovations to Get Smarter
405. Interpreting Big Data: Data Science vs Data Analytics
Data Science and Data Analytics are quite diverse but are related to the processing of Big data. The difference lies in the way they manipulate data.
406. Data Representation Techniques for Efficient Query Performance
Discover how to boost Apache Spark's query efficiency using data sketches for fast counts and intersections in large datasets. Essential for data pros!
407. How to Build a Data Stack from Scratch
Overview of the modern data stack after interview 200+ data leaders. Decision Matrix for Benchmark (DW, ETL, Governance, Visualisation, Documentation, etc)
408. Unlocking the Invaluable Role of Big Data in Modern Supply Chain Management
Let’s take a deeper look at the scale of impact the big data revolution can have for global supply chains and vendor management.
409. "Using this method, I went from a teller to an executive," says Carlo Martinez CEO of Steppingblocks
How I became obsessed with helping students connect college degrees to careers sooner. So, I decided to build a platform and call it Steppingblocks.
410. Let AI Tune Your Database Management System for You
Explore how Reinforcement Learning (RL) is revolutionizing DBMS configuration tuning.
411. Web Scraping API para Extracción de Datos: Una Guía para Principiantes
¿Alguna vez te sucede cuando la gente te pide que escribas una API separada para integrar datos de redes sociales y guardar los datos sin procesar en tu base de datos de análisis en el sitio? Definitivamente quieres saber qué es la API, cómo se usa en web scraping y qué puede lograr con ella. Echemos un vistazo.
412. Manticore Search Now Integrates With Grafana
We are excited to announce that Manticore Search starting from 6.2.0 integrates effortlessly with Grafana.
413. Is Your Latest Data Really the Latest? Check the Data Update Mechanism of Your Database
In databases, data update is to add, delete, or modify data. Timely data update is an important part of high quality data services.
414. How to Use Business Intelligence: 66% of Companies Want to Be More Data-Driven in 2021
How do BI solutions help to make the decision-making process driven by data, improve CX, and speed up reporting? And how can you implement it yourself?
415. Low-Code Development Helps Data Scientists Uncover Analytical Insights
Emerging low-code development platforms enable Data Science teams to derive analytical insights from Big Data quickly.
416. How Much Can You Make as a Data Scientist?
Wondering how much data scientists make? We're here to help you find out about salaries in Data Science and how they are influenced by various factors.
417. How to Use Node Streams to Transform the Largest POI Database
OpenStreetMap (OSM) is maybe the most extensive open data project for geo-data. It has rich information on points of interest (POIs), such as apartments, shops, or offices, globally.
418. Elasticsearch VS Apache Doris in Log Analysis
Discover how Apache Doris revolutionizes log analysis. From schema-free support to cost-effective storage, learn how to build an efficient log analysis system.
419. How Big Data is Keeping Employees Engaged in the Age of WFH
Big data is beginning to emerge as a key tool for businesses to successfully operate on a WFH basis.
420. Data Location Awareness: The Benefits of Implementing Tiered Locality
Tiered Locality is a feature led by my colleague Andrew Audibert at Alluxio. This article dives into the details of how tiered locality helps provide optimized performance and lower costs. The original article was published on Alluxio’s engineering blog
421. Artificial Intelligence: Multimillennial Data Transmitted To Machines With Brains
We are gradually encoding human knowledge in seas of annotated data
422. An Intro to SQL for Data Scientists
The importance of SQL and how to go about learning it
423. Universal Data Tool Update: On-Premise Data Labeling
If you haven’t heard of the Universal Data Tool yet, it’s an open-source web or desktop program to collaborate, build and edit text, image, video, and audio datasets with labels and annotations.
424. How Big Data Can Help Personalize Your Ecommerce Store
Data is everywhere. Every single detail you have ever provided online – from your address to the advertisements you’ve clicked on –is stored by browsers and applications.
425. How To Query JSON in Couchbase via Collections and Scopes
This week I’m attending the 3-day Couchbase Connect event and will be reporting on some of the topics that I find most interesting.
426. 7 BFSI Trends in 2022: Big Data, Blockchain, and More
BFSI sector is anticipated to witness major trend changes in the technology segment. The article will present details regarding the upcoming transformations.
427. Social Network Big Data Will Boost Website Traffic
The importance of social media in business marketing cannot be overlooked. All you have to do is find the best ways to make the best use of it. One such important way to boost your website traffic easily through your social networks is by transport planning and using big data.
428. 'At the Coalface of Implementing Data Stacks': kleene's Co-founder & CEO Andrew Thomas
2-minute look at the building of kleene.ai through a founder's eyes.
429. Predictive Analytics and You: Stagnation by Design
This story begins and ends with algorithms, those series of functions so mathy and boring that rather than think about them at all, most of us would prefer listening to our nine-year-old nephew rattle off a list of his 255 most-favorite Pokemon, organized from most to least interesting.
430. The Direct Lake Mirage: What Really Happens at 99 Million Rows
A real 99M-row benchmark reveals why Import Mode still outperforms Direct Lake in Microsoft Fabric and what the engine truth means for your BI architecture.
431. How AI Has Enhanced Sentiment Analysis Using Product Review Data
Customer feedback is great. But have you been able to turn that feedback into meaningful customer insights? A few years back, brands depended on surveys to gauge customers’ feelings about how their products were performing.
432. How to Gather Actionable Customer Data With Social Media
Before you can start finding things out about your audience, you have to figure out what you want from your social media marketing strategy.
433. All About Parquet Part 01 - An Introduction
Discover Apache Iceberg with a free guide, crash course, and video playlist. Learn efficient data management and processing for big data environments.
434. The Importance of Performant Data Processing Architecture in Creating AI Chatbots
If you believe in the idea that big data is the fuel for artificial intelligence, you will see how important a performant data processing architecture is for us
435. Revolutionizing Data Management for Strategic Decision-Making
With AI and ML creating new datasets that previously didn't exist, the challenge lies in continuously developing skills to visualize this data.
436. Native Analytics On Elasticsearch With Knowi
Table of Contents
437. Beyond Excel and Google Sheets: Innovations in Handling Massive Spreadsheets Amidst the Big Data Rev
We spoke to Jason Hines, the Co-founder of Gigasheet, a big data spreadsheet platform...
438. How Big Data Can Bring Transformative Improvements to Medical Care
In the healthcare landscape, providers and lawmakers alike are faced with the challenge of making the best possible decisions for patients and the industry as a whole. From choosing the best treatments to using resources in a responsible manner, medical leaders are making decisions on a daily basis that can significantly impact health outcomes and costs.
439. Eco-Big Data Applications in the City: Cleaning Up with IoT and ML
Digitalization is possible not only in enterprises. Digital transformation is catching up even with cities to make them more convenient for residents and less harmful to the planet. How to quickly monitor garbage cans, the state of forest parks, cycling and air purity with the help of big data, machine learning and the Internet of things?
440. Make Big Data More Manageable with Smart Sampling
Learn the limits of clustering compression, appreciating Fast-Coresets' efficiency and questioning how fairness and optimality fit into future coreset research
441. Leveraging AI for Insights-Driven Organizational Efficiency Gains
With modern-day work largely centered on digital platforms, automating the handling of big data has become more important than ever. This is where Artificial Intelligence (AI) comes in— performing tasks more efficiently by imitating our abilities to learn and solve problems. As technology advances at breakneck speed, fueled by the IoT environment, it has paved the way for a synergistic relationship between Artificial Intelligence and Big Data.
442. The Importance of Monitoring Big Data Analytics Pipelines
In this article, we first explain the requirements for monitoring your big data analytics pipeline and then we go into the key aspects that you need to consider to build a system that provides holistic observability.
443. Data Journalism 101: 'Stories are Just Data with a Soul'
Gone are the days when journalists simply had to find and report news.
444. From Investment Banking to DaaS: My Journey of Understanding Financial Struggles as a Gen Z
The future of personal finance for Gen Z is data-driven. With Syval, users will have the chance to reflect on every financial decision they make.
445. Don't Let Them Fool You: Manipulative Strategies Used By Big Tech Companies To Sell You Stuff
Do you know how your apps work? Are you aware of what tech companies are doing in the back with your data? And what’s more revealing: do you know which of your action are actually influenced by those apps? When you take a trip with Uber, buy stuff on Amazon, or watch a movie on Netflix: when are you consciously deciding and when are you being heavily influenced?
446. How to Build or Transform Recruitment Platforms Using Web Data?
Looking to build the best recruitment platform ever? Here are 10 things you should keep in mind.
447. Can Big Data Solutions Be More Accessible And Affordable?
Below you can find the article of my colleague and Big Data expert Boris Trofimov.
448. How to Democratize Access to Data Insights for Businesses of All Sizes
Messy government data has been part of the reason we've been unable to understand the COVID-19 pandemic. If federal organizations can't decode big data, what hope do small businesses have?
449. Is Your Business Ready for AI Implementation?
Artificial intelligence (AI) and machine learning are no longer futuristic theories. They are now real technologies with real applications in numerous businesses. The Forbes Insights poll, together with Dell Technologies and Intel, showed that AI is a key component of digital development, but only a quarter of Chief Experience Officers surveyed say they have implemented these technologies in their company. What is the reason for such low AI penetration in organizations and is your company ready to use machine learning? In this article, we will share our thoughts on the impact of AI on business and how to implement it faster.
450. Want To Earn 100k and Above? Then Look to Data Science Jobs
Data science is more vital than ever in the AI era, offering high salaries and essential skills for tech professionals.
451. Data Is Now a Luxury Good: Here’s Why (It Shouldn’t Be)
452. Why Re-invent the Wheel? Use Past Workloads for Smarter DBMS Tuning
Use historical workload data and similarity techniques to speed up DBMS tuning, reduce training time, and optimize configurations more efficiently.
453. How Big Data Is Disrupting Big Business Right Now
Image Credit: Unsplash
454. The Data Lakehouse Isn’t the Silver Bullet Teams Think It Is
A data engineer breaks down why lakehouse architecture isn’t the revolution it’s marketed as—and why data modeling, quality, and ownership matter far more.
455. Data Collaboration: Challenges and Applications in Business
Data is the future of business. Being able to effectively collaborate at the level of data operations is critical to business success.
456. The Hidden Flaw in Real-Time Fraud Detection (and the Hybrid Solution That Works)
Real-time fraud detection requires both speed and accuracy - hybrid event-based aggregation delivers both.
457. How the Future of Automation Will Drive Innovation
Automation is an exciting prospect. Who doesn’t like the idea of having menial tasks completed quicker and more effectively than they could have been by a human?
458. Open Source is the Only Way to Address the Long Tail of Integrations
Wouldn’t it be great to bring the time needed to build a new data integration connector down to 10 minutes? This would definitely help address the long tail of
459. What Is the Best Way To Compress Big Data Without Sacrificing Accuracy?
Fast-Coresets balance speed and accuracy in clustering compression, outperforming uniform sampling but requiring careful dataset selection for optimal results.
460. A Day in the Life of a Data Scientist at a Climate Change Startup
A guided tour into the life of a data scientist at a climate-tech startup.
461. Data Organization – The Great Differentiator in the Digital Era
In business, efficient processes can make or break an organization. If processes are not executed properly, companies lose time, money, and damage their reputation.
462. Pruning Techniques for Reducing Workload and Configuration Space Complexity
Pruning techniques like feature projection, ranking, and reduction help reduce DBMS tuning complexity, improving efficiency in data collection and training.
463. The Key to Solving DBMS Tuning Problems
Explore the challenges of DBMS parameter tuning and discover solutions like Bayesian optimization, neural networks, and reinforcement learning.
464. Crypto Use Explodes, Data Will Help Investors Make Better Decisions
Investors need good data to make good decisions, and new AI platforms will provide deeper analysis
465. When 125 Million Item-Locations Lied To Us: What Retail Forecasting Taught Me About Data Truth
Retail forecasting fails when item-location data disagrees across systems; rebuilding a single source of truth restores accuracy and trust at scale.
466. Providing Next Generation Customer Experience with Sagi Eliyahu, CEO at KMS Lighthouse
This article talks about how artificial intelligence and machine learning tools are used to improve and automate customer experience with automated smart reply.
467. How to Tell the Difference Between Data Warehouses, Data Lakes and Data Lakehouses
Struggling to harness data sprawl, CIOs across industries are facing tough challenges.
468. Startup Interview with Zoltan Csikos, Co-Founder & CEO, Neticle
Neticle offers a range of text analytics tools for businesses. If you have textual data to analyze, Neticle has a solution for you!
469. Big Data: 70 Increíbles Fuentes de Datos Gratuitas que Debes Conocer para 2020
Por favor clic el artículo original：http://www.octoparse.es/blog/70-fuentes-de-datos-gratuitas-en-2020
470. Runtime-Based Workload Characterization in DBMS Tuning
Learn about runtime-based workload characterization in DBMS tuning, including metrics like query cache utilization, memory allocation, and locking overhead.
471. How Big Data Can Help Build Biotech Products
New methods and discoveries, such as next-generation genome sequencing, generate vast amounts of data and transform the scientific landscape.
472. Hacking Your Way to Being an All-Star [Infographic]
What does it take to make a team leader who pulls a team together? How do these qualities lead a player to become a strong contender for the NBA All-Stars team? Great basketball players know their teammates’ strengths and weaknesses and they understand how to play to every player’s strengths to make the team stronger as a whole. By setting a good example and remaining optimistic about the team as a whole, Tobias Harris has proven his value as a team player to the 76ers.
473. Demystifying Dimensional Modelling: Unveiling the What, Why, and Who's
An Introduction to the art and science of dimensional modeling with relational databases
474. The 5 Ingenious Data Structures (and What They Actually Do)
Explore 5 advanced data structures that go beyond arrays and linked lists. Learn how B-Trees, Bloom Filters, Radix Trees, and more power modern systems.
475. Should Organizations Hire Data Ethicists?
Although AI systems are advancing rapidly, they still produce skewed results. Read on to learn the steps organizations might take to clear the unconscious bias.
476. Crowdsourcing AI Training Data — A Discussion With William Simonin, Ta-da Chairman
A conversation with Ta-da's William Simonin on leveraging crowdsourcing to collect AI training data, plus tackling AI challenges and solutions.
477. The Advantages of a Hybrid Deployment Architecture
See how a hybrid architecture marries the best of the SaaS world and on-prem world for modern data stack software.
478. How Advanced Analytics Can Improve the Public Sector
Advanced analytic models can identify and predict negative outcomes such as health and safety challenges or compliance risks that would be overlooked by manual.
479. Indoor Positioning and Predicting the Most Suitable Boutiques in Shopping Malls for Customers
Indoor navigation and machine learning combination both for helping users to find the most suitable stores and for helping stores to advertise their products.
480. Cloud Solutions Propelled Into The Spotlight Courtesy Of Covid-19
In the wake of the COVID-19 pandemic, cloud service solutions have been thrown into the limelight as companies and organisations across the globe grapple with the rapid shift to remote working and learning. With the widespread closure of non-essential organisations and businesses forcing organisations’ leaders to consider new and innovative approaches to shifting their businesses online, the move to cloud computing has become a far greater priority than ever before. The industry statistics demonstrate this: according to new figures from analyst firm Gartner, by the end of 2020 we will have seen the global public cloud services market reach $266.4 billion, up from $227.8 billion in 2019.
481. How Modern GTM Systems Drive Revenue Growth: Bridging Business Strategy with Technology
Modern GTM strategies have evolved into data-driven machines. Discover how aligning data, technology, and RevOps creates predictable, scalable revenue growth.
482. 4 Critical Steps To Build A Large Catalog Of Connectors Remarkably Well
The art of building a large catalog of connectors is thinking in onion layers.
483. Understand Data Analytics Framework Using An Example From General Electric Company
The framework will allow you to focus on the business outcomes first and the actions and decisions that enable the outcomes.
484. Automatic Configuration Tuning on Cloud Database: A Survey
Discover state-of-the-art techniques in automatic parameter tuning for cloud database management systems (DBMS)
485. How Configuration-level Pruning Reduces Optimization Time in DBMS Tuning
Learn about configuration-level pruning techniques for DBMS tuning, including SRCC, PCA, LASSO, Random Forest, and HeSBO.
486. Unlock Smarter DBMS Tuning with Neural Networks
Explore how Neural Network (NN)-based solutions are transforming DBMS configuration tuning.
487. Maximizing E-commerce Potential with Refined Data Analytics and Storage Architecture
In this post, I write about how my team carries out refined operations, based on our own Data Management Platform (DMP).
488. Bringing C++ to Query Execution: Why the Future of Data Engines Is Native
A deep dive into why query engines are moving to C++ and how Velox delivers faster, more predictable execution for systems like Presto.
489. Tuning DBMS: The Secret Life of Queries and Runtime Data
Explore the two main aspects of DBMS workload characterization: query-level and runtime-based.
490. The Unending Data Dilemma: Navigating Privacy, Breaches, and Regulations
Overview of data privacy, laws, and regulation
491. Web Data Has Become a Commodity and Needs a Marketplace to Grow
Web scraping and web data is a commodity, and we need a market to trade it.
492. Accelerating Innovation: How Covid Has Prompted Technological Evolution Within Healthcare
Let’s take a deeper look into some of the most significant tech innovations that have been prompted by the emergence of Covid-19.
493. Do You Need All This Data?
A “lean data” strategy is necessary for today’s e-commerce businesses to stay nimble, avoid “data muck” and not be bogged down by too much data.
494. Data Preparation: The Case for Using Automated, ML-Based Tools
Data preparation has always been challenging, but over the past few years as companies increasingly indulge in big data technologies, data preparation has become a mammoth challenge threatening the success of big data, AI, IoT initiatives.
495. Solutions for Upgrading Apache DolphinScheduler from Version 1.3.4 to 3.1.2
This article primarily records the issues encountered during the upgrade process and aims to help others facing similar challenges.
496. Buckle Up and Enjoy Some Graph Therapy
Graph Therapy. The Year of the Graph Newsletter, June / May 2020
497. The eCommerce Turn of The Art Market: Trust, Transparency, And Trustworthiness
Now that the online art marketplaces are finally going mainstream, how can the experience be matched to other online marketplaces? Data might be the key.
498. How to Fix Sqoop Not Found and ClassNotFound in DolphinScheduler
Using DolphinScheduler with Sqoop can streamline data synchronization across systems. But beginners often run into frustrating errors during setup and execution
499. Auto-Increment Columns in Databases: A Simple Trick That Makes a Big Difference
An introduction to auto-increment columns in Apache Doris, usage, applicable scenarios, and implementation details.
500. Prune Your Way to Better Database Management System Performance
Learn about runtime-based workload characterization in DBMS tuning, including metrics like query cache utilization, memory allocation, and locking overhead.
Thank you for checking out the 500 most read blog posts about Big Data on HackerNoon.
Visit the /Learn Repo to find the most read blog posts about any technology.