Let's learn about Data Management via these 110 free blog posts. They are ordered by HackerNoon reader engagement data. Visit the Learn Repo or LearnRepo.com to find the most read blog posts about any technology.
Data management is the practice of organizing, storing, and maintaining data effectively and securely throughout its lifecycle. It ensures data quality, accessibility, and compliance, which are crucial for informed decision-making and operational efficiency.
1. The High-Frequency Trading Developer’s Guide: Six Key Components for Low Latency and Scalability
High-frequency trading (HFT) relies on complex algorithms to profit from small price discrepancies, requiring ultra-low latency and high-speed order execution.
2. Applying Transitive Closure to Sort Products Into Categories, Considering Nesting and Overlaps
A guide to efficiently managing nested categories and overlapping products, ensuring fast retrieval without duplicates in e-commerce systems.
3. How to Build an End-to-End ML Platform
In this paper, readers will find an overview roadmap to generating a strong ML system that starts from data management to streamline operations efficiently.
4. Don’t OFFSET Your SQL Query’s Performance
To implement pagination without unexpected performance issues on large sets of data, use "WHERE id > N" instead of "OFFSET N”.
5. Listing of JavaScript Editable Table Libraries
In this article, we review a list of commercial JavaScript editable table libraries and UI widgets that can be incorporated into web applications.
6. How To Process Engineering Drawings With AI
Learn why ready-made AI tools are not well-suited for engineering drawings processing and how to actually use AI to extract data from technical drawings.
7. The Snowflake Hack and Its Domino Effect
Learn how to secure your company's data in the wake of major breaches. Discover a four-zone approach to data management that balances security and accessibility
8. Effortlessly Launch LangChain APIs with LangServe and MinIO Integration
Streamline LangChain app deployment with LangServe and MinIO, creating powerful, production-ready APIs for seamless data management.
9. How to Build a Winning Proposal for a Data Quality Project
Build a winning data quality project proposal with clear goals, strong justification, and proven strategies that secure leadership approval and drive success.
10. Streamlining Form Validation with JSON Schema for Front-End and Back-End
Efficiently validate forms with a single set of rules for both front-end and back-end using JSON Schema.
11. Build vs Buy: What We Learned by Implementing a Data Catalog
Why we chose to finally buy a unified data workspace (Atlan), after spending 1.5 years building our own internal solution with Amundsen and Atlas
12. #NoBrainers: You Need A High Performing Low Latency Distributed Database
13. Why My Brain Is Wired for Clojure
The world according to Clojure is nothing but data.At any point in time, it’s one piece of data.
14. Infinite Scrolling vs Pagination: Making the Right Choice for React Apps
Though infinite scrolling & pagination offer viable options for handling large datasets in React applications, check here what suites you best.
15. How X (Formally Twitter) Slashed Costs by 60% Without Sacrificing Quality
A case study on Twitter's remarkable cost reduction through cloud exit and its implications for businesses.
16. Self-Sovereign Identity Systems: How Businesses Win From Letting Go of Customers’ Data
In the late 2010s, “the data rush” ushered businesses to collect as much user data as possible in the high hopes of making their products, marketing and sales processes more effective.
17. Avoiding the Pitfalls of Data Mesh Adoption
Chefs cook data in decentralized kitchens, but beware! Lack of training, clarity, & governance can turn your feast into a Kitchen Nightmare.
18. Why PostgreSQL Is the Bedrock for the Future of Data
Explore the rise of PostgreSQL as the de facto database standard, its impact on software development, and the key trends driving its widespread adoption.
19. Advancing User Data Governance with Data Lineage
This article will discuss how data lineage can help in user data governance and explore how serverless technology can be incorporated to achieve better results.
20. NoSQL vs SQL: Comparison From the Development Team
Let’s dive into the main types of databases, their main features, and their working principles. And also development differences to be more practical.
21. Data Preparation for Machine Learning: A Step-by-Step Guide
Many businesses assume that feeding large volumes of data into an ML engine is enough to generate accurate predictions.
22. 22 Best Tools to Use for Marketing Startups in 2022
Good advice on the useful tools in different marketing niches. Short description of tools that can be really useful for marketing startups
23. Data-Driven Advertising and Its Impact On Our Privacy-Driven World
Do we actually need so much data to do effective marketing?
24. The Growth Marketing Writing Contest by mParticle and HackerNoon
mParticle & HackerNoon are excited to host a Growth Marketing Writing Contest. Here’s your chance to win money from a whopping $12,000 prize pool!
25. 5 Software Documentation Tools That Make Managing Technical Documentation Easier
Software documentation is an important step in the process of software development. These are the 5 tools that make the cut when it comes to functionality.
26. The Failed Promises of Extract, Transform, and Load—and What Comes Next
Faster, Better Insights: Why Networked Data Platforms Matter for Telecommunications Companies
27. Convert Formatted Text Into a Data Structure Using Parsing
Parsing is a process of converting formatted text into a data structure. A data structure type can be any suitable representation of the information engraved in the source text.
28. SQL Server New Features and Release Date
SQL Server 2022 has received a lot of attention since its release. What are the new features we explore in the latest SQL Server version? Read on!
29. Best Practices For Backend Data Security
Backend data security relies in encryption, access control, data backup and other such features to exist. These best practices are intended for the backend.
30. Building a Data Management Strategy: Importance, Principles, Roadmap
Already routinely called the currency, the lifeblood, and the new oil of the modern business world, data promises organizations unbeatable competitive advantages.
31. Why Data Governance is Vital for Data Management
Both data governance and data management workflows are critical to ensuring the security and control of an organization’s most valuable asset-data.
32. How To Use Change Data Capture for Fraud Detection
Still relying on overnight processes to drive your decision making? Maybe it’s time to consider an evaluation of your CDC pattern that uses new technology.
33. How RAG Improves Database Management
RAG is transforming database management with accurate retrieval, real-time insights, and natural language querying to help teams manage and understand data inte
34. The Hidden Cost of Bad Data: Why It’s Undermining Your AI Strategy
Poor data quality is undermining your AI strategy. Uncover the hidden costs and follow our roadmap to transform bad data into a high-ROI strategic asset
35. The Role of Ontologies in Data Management
Ontologies organize data, enhance interoperability, and drive insights across domains with structured frameworks.
36. Spreadsheets Don't Scale Well: Here's How 3 Startups Have Overcome Those Limitations
Spreadsheets are the versatile go-to for a wide array of business practices. Companies use them for everything from finance to marketing analytics. "Just use Excel" has even been the go-to battle cry of data scientists who are frustrated by watching companies waste billions of dollars trying to ramp up analytics programs that they're not ready for.
37. What Is a Data Management System and Why Your Business Needs It
Data is king, but raw data needs management to unlock its value. Businesses need data management systems to gain insights, improve processes, and drive growth.
38. Data Management and Consolidation in the Integration of Corporate Information Systems
Explore ETL and Kafka Streams architecture for advanced data management and real-time processing and analytics in corporate systems.
39. Data Observability: The First Step Towards Being Data-Driven
In a nutshell, data reliability is a BIG challenge and there is a need for a solution that is easy to use, understand, and deploy, and also not hea
40. Building an Efficient AI Platform for Data Preprocessing and Model Training
Lei Li, AI Platform Lead, and Zifan Ni, Senior Software Engineer from Bilibili, share how they increased the training efficiency on their AI platform.
41. HarperDB is More Than Just a Database: Here's Why
HarperDB is more than just a database, and for certain users or projects, HarperDB is not serving as a database at all. How can this be possible?
42. Intro to Data Vault Modeling: Agility, Scalability, and Practical Applications Explained
The practical use of Data Vault models, as illustrated through querying customer orders and analyzing product sales, demonstrates the methodology's flexibility,
43. What is Data Profiling? Concepts and Examples
Learn the concepts of data profiling and how it can speed up the debugging the quality related incidents across the data stack.
44. Mastering Scraped Data Management (AI Tips Inside)
Let's explore a few techniques to handle scraped data, including automatic data processing via AI.
45. 'Experience is a Double-edged Sword': Kyle Kirwan, CEO of Bigeye
An interview with the founder and CEO of Bigeye, a data observability platform.
46. Understanding Data Lineage: Key Strategies for Ensuring Data Quality and Compliance
Data lineage refers to the process of tracking data from its origin to its destination, including all transformations and movements in between. It is crucial fo
47. From Data Chaos to Clarity: The Data Marketplace vs. Data Mesh
Unleash the power of data! Explore the Data Marketplace, an alternative to Data Mesh that prioritizes collaboration, clarity, and value for informed decisions
48. Efficient Log Analysis: Harnessing the Power of Regex with BindPlane OP and OpenTelemetry
Deciphering Complex Logs with Regex
49. "We want functional decentralization" Q&A with Wildland Creators
Wildland is a new, open data management protocol with improved users' privacy, security, and multi-categorization. A Q&A with J. Zawistowski and A. Regulski.
50. Is Your Organization Truly Data-Driven? A 5-Point Checklist
Most C-level executives I've met aspire for their organization to be data-driven, but when probed on what exactly do they mean by "being data-driven," their answers fall short of their vision.
51. 15 Best Project Management Tools
Project management systems are supposed to make the life of teams easier and the work process faster and more efficient.
52. Mastering NumPy Arrays(Part 1): Stacking and Splitting
A comprehensive guide for NumPy Stacking. How to stack numpy arrays on top of each other or side by side. How to use axis to specify how we want to stack arrays
53. How to handle your startup data like a big tech
Core principles in data management that all big tech companies adhere to can and should be adopted by startups.
54. Transforming Data Management: How ibml is Revolutionizing Information Capture and Processing
Businesses are rushing to implement AI solutions, but may overlook the quality, accuracy, and accessibility of data they are using.
55. How Distributed Databases Power Mission-Critical Business Apps: A Case Study with Amey Banarse
Explore how Amey Banarse uses distributed databases to power mission-critical apps for top companies, enhancing scalability, cost-efficiency, and performance.
56. Mastering Stream Processing: A Guide to Windowing in Kafka Streams and Flink SQL
An introduction to windowing for better analysis of events in streaming technologies like Kafka Streams and Flink.
57. Vector Databases: Unlocking New Dimensions in Similarity Search
Explore how vector embeddings and LLMs transform search capabilities in streaming platforms and enhance user discovery and personalization.
58. Data Management in 2024: Will Open Data Formats Shape a “Sixth Platform”?
Can open data formats lead to a best-of-breed data management platform? It will take Interoperability across clouds & formats, as well as semantics & governance
59. How AI-Powered Data Mapping is Democratizing Data Management
Learn how AI-powered data mapping is transforming data management, making it more accessible and efficient for everyone.
60. From Raw Data to Actionable Insights: The Power of Data Aggregation
This article examines data aggregation processes: collecting data to present it in summary form.
61. Predicting the Future: Using Machine Learning to Boost Efficiency in Distributed Computing
Learn how Machine Learning boosts Distributed Computing efficiency by predicting workloads, optimizing resource allocation, and driving sustainable data centers
62. DevOps + Data: DevOps for Data Management
Fresh Approaches to Effective Data Management.
What if we applied the existing DevOps techniques and patterns to the management ?
63. Closed Source vs Open Source PIM: Navigating Regulatory and Scalability Challenges
Closed Source Vs. Open Source PIM Comparison
64. Generative AI: 3 Topics to Learn as a Data Engineer in 2024 and Beyond
Discover the top three areas data engineers can learn to leverage generative AI in 2025.
65. Model Paradigm for Engineering
66. Why Data Governance in Healthcare Matters in 2024 With Nithin Narayan Koranchirath
An interview with Nithin Narayan Koranchirath about the power of data governance in healthcare, protecting patient privacy, and improving outcomes.
67. Data Potential: 10 Reasons Apache Iceberg and Dremio Should Be Part of Your Data Lakehouse Strategy
Discover the powerful synergy of Apache Iceberg and Dremio, revolutionizing data management and analytics.
68. How a Shenzhen Smart Factory Uses Apache DolphinScheduler to Orchestrate Industrial Data
A leading Shenzhen manufacturing enterprise uses #ApacheDolphinScheduler to standardize data pipelines and deploy dozens of factories in a single day.
69. The Pillars of Data Governance and Why They Matter
Data Governance 101: How organizations protect data, maintain a golden copy, and stay compliant through quality, stewardship & access control.
70. Creating a Dependable Data Pipeline for Your Small Business
In this article, I will be showing you how to build a reliable data pipeline for your small business to improve your productivity and data security.
71. Using Data Analytics for Unhindered Business Growth
72. 5 Data Management Principles That Matter in 2021
Let’s consider a few fundamental data management principles that matter. Data management is less about filing information and more about finding order.
73. 5 Features To Consider When Looking For a Reliable Data Loss Prevention (DLP) Software To Buy
As data loss prevention (DLP) solution plays a crucial role to prevent unauthorized access to an organization’s sensitive data.
74. In-Depth Guide to Plugin Architectures with Spring, Consul, and Camel
For the past couple of months I've been working on a data management tool I'm calling OpenDMP. As I've started adding more features, I've run into a scalability issue a bit sooner than I had expected and so I decided to tackle what is hopefully the biggest remaining piece of the project's system architecture.
75. Processes and New Technologies in Data Transformation
In this article, I explore the benefits, types, and processes of data transformation and how it contributes to data management, integration, and new technologie
76. How To Manage Sensitive Data Using SQL Data Discovery and Classification
The 17.5 version of SQL Server Management Studio (SSMS) brought with it a new built-in security tool. Since then, the Data Discovery and Classification feature has become a difference-maker in the protection of sensitive information.
77. What is a 'Data Fabric'?
A Data Fabric is a mix of architecture and technology that aims to ease the difficulty and complexity of managing several different data types.
78. A Leader's Guide to Data-Driven Success
Transform data from a source of frustration into a powerful business tool with this practical guide for executives.
79. I Stress-Tested 5 Data Catalogs With Real Governance Scenarios. Most Failed Silently.
"Governance is a process problem wearing a tool costume." I tested 5 data catalogs against real data incidents. Here is what actually broke.
80. Welcome to the Multimodal AI Era
Explore the rise of multimodal AI, a new frontier in artificial intelligence that integrates text, images, audio, and video for a more holistic approach.
81. How Blockchain Can Simplify Post-Merger Master Data Management
82. From Data Mess to Data Mesh: How to Optimize Business Intelligence
Digitization as a trend means the world is now generating more data than ever before. How said data is managed is crucial for business and individuals alike.
83. Revolutionizing Data Management for Strategic Decision-Making
With AI and ML creating new datasets that previously didn't exist, the challenge lies in continuously developing skills to visualize this data.
84. Strategies and Best Practices for Ensuring Data Consistency
85. Managing the Spatial Data for My Wildfire Detection Dashboard
A look at how I manage the spatial data for a wildfire detection dashboard that I previously built.
86. Getting Started With Apache Iceberg and Resources if You Would Like To Go Further
Discover how Apache Iceberg revolutionizes data lakehouse architecture with efficient table management and powerful features like schema evolution.
87. Data Reliability in an Unreliable World
What is common between streaming movie on Netflix, searching air ticket on Google, buying clothes on Amazon? You rely on distributed computing to do it.
88. Applying Criminology Theories to Data Management: "The Broken Window Theory: and "The Perfect Storm"
What can be done to prevent “Broken Windows” in the primary data source? How can we effectively fix existing “Broken Windows"?
89. Telemedicine App For More Accurate Identification of Severe Brain Conditions
This text is about a robust app integrated with measuring devices, which identify brain conditions such as Parkinson’s, Alzheimer’s, and ADHD
90. Talking Less and Doing More - Interview with Startups of the Year Nominee, Comquest
Comquest Software has been nominated in HackerNoon's annual Startup of the Year awards in Telangana, India. Here's why.
91. Nevermined: How organizations can manage or monetize their data with a next level solution
This post provides a short technical overview of Nevermined’s capabilities
92. How to Copy a Column From One Sheet to Another in Google Sheets Easily
Whether you're a Google Apps Script Developer or just someone looking to streamline their workflow, this script will be a helpful tool for you.
93. Data Masking: How it Can be Implemented Correctly
94. ScyllaDB Hits Fourth Generation with Raft, Tablets, and a Cloud-First Vision
When ScyllaDB started, the goal was to be the fastest NoSQL database available in the market. However, raw speed does not necessarily make a good database.
95. The Unending Data Dilemma: Navigating Privacy, Breaches, and Regulations
Overview of data privacy, laws, and regulation
96. Auto-Increment Columns in Databases: A Simple Trick That Makes a Big Difference
An introduction to auto-increment columns in Apache Doris, usage, applicable scenarios, and implementation details.
97. Effective Adoption of Data Warehouses in Healthcare: A Complete Guide
Healthcare providers deal with a lot of data. This data is often stored across a variety of legacy systems that don't communicate with one another that well. Not only do data discrepancies eat up medics' time (think: nine hours per week), but they also influence the quality of care. You know it better than anyone: drawing a complete picture of what a patient has and is experiencing health-wise is the first step toward correct diagnosing and effective treatment.To fend off healthcare data disparities, medical organizations have long been turning to data management and data analytics providers. The aim? Bring siloed data together into single, consolidated storage — a healthcare data warehouse — and use it to draw insights.This blog post covers vital aspects of adopting a data warehouse in healthcare, zooming in on its technical characteristics, highlighting the value a centralized data storage can drive for medical organizations, and providing a high-level data warehouse implementation roadmap.
98. Is Your Business Suffering from Big Data Burnout? 5 Ways to Democratize Data
With so much data available at your fingertips, if you fail to implement a strong system, your business is at risk of suffering from big data burnout.
99. Your Customers Don't Care About Your Data Strategy — Until It Fails Them
Customer experience failures often stem from poor data governance. Learn how trusted, accurate data powers AI, personalization, and digital trust.
100. How Aging Data Becomes a Crisis
A real-world story of spotting data retention risk early and turning SQL analysis into a governance framework before it became a crisis.
101. How To Drive Business Value through Smart Data
102. The blockchain missing link in remote project management.
103. A Primer on Decoupling SQL Engines from Hive Data Warehouse
Are you using SQL engines, such as Presto, to query existing Hive data warehouse and experiencing challenges?
104. Data Platform as a Service: A Three-Pillar Model for Scaling Enterprise Data Systems
DPaaS solves the enterprise data scalability paradox with declarative policies, multi-plane architecture, and continuous reconciliation.
105. What You Need to Know About Tabular Data as a Challenge
Despite AI/ML research focusing on unstructured data, tabular data remains the primary area of time and financial investment in the Data Integration world.
106. Serving Structured Data in Alluxio
This article introduces Structured Data Management (Developer Preview) available in the latest Alluxio 2.1.0 release, a new effort to provide further benefits to SQL and structured data workloads using Alluxio. The original concept was discussed on Alluxio’s engineering blog. This article is part one of the two articles on the Structured Data Management feature my team worked on.
107. A New Approach to Solve I/O Challenges in the Machine Learning Pipeline
Training and caching data can be done in a transparent and distributed way to improve training performance and simplify data management.
108. PostgreSQL Couldn’t Handle Our Time-Series Data—TimescaleDB Crushed It
Learn how TimescaleDB's compression features reduced storage needs by 83% while maintaining query performance.
109. Why the Decentralisation of Data Is Crucial in Today's World
Decentralizing data management is the process of collecting, storing, organizing, retrieving, and processing data.
110. Serving Structured Data in Alluxio: Example
In the previous article, I described the concept and design of the Structured Data Service in the Alluxio 2.1.0 release. This article will go through an example to demonstrate how it helps SQL and structured data workloads.
Thank you for checking out the 110 most read blog posts about Data Management on HackerNoon.
Visit the /Learn Repo to find the most read blog posts about any technology.