Most CSV headaches start the moment your data stops being truly flat. CSV is the most common data format in the world. It’s simple, human-readable, and supported by almost every tool. And yet, it’s one of the most heavily misused formats out there. A CSV is built for
We’ve all seen the “10 quick tips for cleaning your data” posts. Most of them leave out the tricky parts — the places where you can quietly ruin your dataset without even noticing. A few that deserve more attention: 1. Format conversion is dangerous. Getting data into the format you
Some tools quietly earn a permanent place in your toolbox. KNIME is one of mine — a visual data workflow builder that’s saved me from endless spreadsheet wrangling and script debugging. KNIME started at the University of Konstanz in 2004 and has grown into a mature, open source platform for
We all know the best practices. Write clean code. Use libraries. Document everything. But here’s the thing: in the real world, things are messy — and I’m no exception. So today, let’s talk about my data engineering sins. Not as a confession, but as a mirror. Because chances
Even the Library of Congress agrees: boring formats win in the long run. In our daily data work, we often treat the data we get as throwaway material. We read it, interpret it, combine it with other data, and finally reach our desired result. Then we toss the intermediate data
Numbers are easy for computers — that’s what we’ve been told. But try storing 9007199254740993 in Excel and you’ll get… a different number. What’s going on? The invisible conversion When I wrote the first version of my CSV editor Tablecruncher, one of my non-negotiables was: Don'
How do you know if two names refer to the same person? That’s the question behind a new EU regulation that will soon require European banks to warn customers during money transfers if the payee’s name doesn’t match the account holder. It’s meant to prevent fraud
Some years ago, I consulted for a company with recurring website issues — broken features, slow performance, inconsistent analytics. After a bit of digging, the root cause became clear: they had layered so many user tracking systems into their site that it had become fragile and unreliable. When I asked why
This is issue #007 of The Missing Header Data problems don’t just happen in IT departments. Even in civil engineering, interpreting data correctly is crucial — and mistakes can cause enormous trouble. In 2003, a bridge called Hochrheinbrücke, connecting Germany and Switzerland across the river Rhine, was built. The project
This is issue #006 of The Missing Header Did you ever get an email with a sentence like: We meet at the café" You probably guessed the sender meant a coffeehouse. But where's this letter garbage coming from? This is a classic symptom of broken character encoding.
This is issue #005 of The Missing Header About a year ago, I asked the AI search engine Perplexity a simple question: “How many otters live in Europe?” The answer? 350. Now, the European otter is endangered in some regions — but not that endangered. There are far more than just
This is issue #004 ofThe Missing Header — formerly known as the Tablecruncher Newsletter* Recently, I got a CSV file with column names like state, amount, and the ever-popular date. Even after looking at the content for a while, their meaning wasn’t obvious. What kind of “date” are we talking