Category Archives: Education

But More Importantly…

climate-changeThose of you who follow my blog know that I’m virulently anti-gun. In fact, I’ll take any opportunity to slip my disdain for guns and the deplorable people who own them into any discussion. Which is why you should definitely go back and read this, and this, and even this.

But not now! Because more importantly… climate change.

As much as I loathe, hate, and despise guns, I fear climate change far worse. No matter what your issue, you are extremely foolish if you do not prioritize climate change far ahead of it. Humanity will survive gun violence, wars, poverty, hate, bigotry, diseases, despots, jobs, slavery, even genocides. But we may likely not survive climate change. Every other issue can be fixed, waited out, and overcome in the long term. Climate change is a death warrant for civilization, for mankind, and possibly for all life on Earth. It’s a terminal disease, game over, if not treated with every means we can muster and more.

So how can you ever rationally argue that efforts to curb climate change must wait because your issue, however important, is more urgent and existential? And no, we cannot “do both.” We must still prioritize. If we spend effort on your issue or even my issue then we are not doing enough to avert catastrophic climate change.

Most of my readers have to know that I’m an outspoken atheist activist. However, I cannot prioritize my atheist movement over climate change. Not even remotely. In fact, if atheists are indeed the more rational and sensible humanists that we think we are and claim to be, we should be taking a leading role in battling climate change. Sadly my atheist community as a whole is not showing such wisdom and leadership.

If there is one litmus test in the next Presidential election, it should be climate change. Not abortion, or gender equality, or a Wall, or fealty to Capitalism, or anything else… because more importantly, climate change.

In a recent interview Presidential candidate Pete Buttigieg rattled off ten or so things he would prioritize as President. Not one was climate change. When asked about climate change, he made a dutiful perfunctory comment about it. This should disqualify him utterly. Even if he does make stronger comments about climate change later, I would have no confidence that he is sufficiently sincere.

In fact, at this time, the ONLY candidate we should be strongly considering is Washington State Governor Jay Inslee. He is the only candidate showing the intelligence, leadership, and long-term thinking that we literally cannot live without. Others might make progress on health care, or immigration, or jobs, or LGBTQ rights. But really, will any of that ultimately matter if we fail to mitigate the worst impacts of catastrophic climate change?

Here’s what you should do. Ask your candidates at all levels about what they will do about climate change and make it an unequivocal priority. Be willing to put aside your own issues in order to work together to make progress on climate change. Demand that the social and religious organizations that you affiliate with push for action on climate change.

And finally, in the signature line of your emails, add the line “But more importantly climate change.” This will remind both you and your recipients that while whatever we are discussing is important, it does not begin to compare with climate change.

 

Religion in Public Schools

The teaching of religion in public schools is a topic that stimulates a great deal of honest debate on all sides of the issue. Should religion be taught at all? And if so, what religions? Even well-meaning atheists might feel that religion should be taught, as long as all religions – and atheistic perspectives as well – are taught equally and fairly without bias.

That sounds laudable and enlightened in theory. However, many plans that sound great in theory inevitably turn out to be disastrous when put into practice. Teaching religion in public schools is one such example.

I have personal experience with this. While serving in the Peace Corps in South Africa, I worked for their Department of Education. The South African Constitution requires that all religions be treated equally. In order to comply with the spirit of their Constitution, the Department of Education has adopted a policy that all religions should be taught fairly and equally in the public schools.

Sounds great right? The trouble is that teachers, particularly rural teachers, do not know all religions and do not care to know all religions – let alone teach them fairly. At the point where lofty policies touch the students. all that this accomplishes is to give teachers cover to preach and proselytize their own religious views in the classroom and to misrepresent and disparage all other religions – and atheism is demonized most of all.

The problem of state sanctioned religious instruction is not merely a matter of the recruiting and training and monitoring of teachers. False even-handedness spills over into teaching materials as well. Science texts typically enumerate a long list of native creation myths as legitimate. In at least one science text, after describing the monkey myth, and the milk myth, and many others, it concluded with what was almost an obligatory footnote that said “and some scientists believe that the world was created by natural means and human beings evolved.”

This sort of false balance, not unlike giving equal deference to climate change deniers, is an almost inevitable consequence of a misguided and ill-fated attempt to be fair and inclusive with regard to the teaching of religion.

I came away from my experience in South Africa more convinced than ever that our American system of simply keeping religion out of our public schools is on balance the best, most practical system of fairness. There is no shortage of alternate venues where people can preach and teach religion as much as they wish. Therefore, there is no compelling need being met by including religion in public schools, that warrants the certain risk of abuse and unintended consequences.

Assiduously keeping religion out of our public schools is in fact the more fair, the more enlightened, and the more realistic policy position.

Our Amazing Yet Deeply Flawed Neural Networks

NeuralNetwork

Back in the 1980’s when I did early work applying Neural Network technology to paint formulation chemistry, that experience gave me fascinating insights into how our brains operate. A computer neural network is a mathematically complex program that does a simple thing. It takes a set of training “facts” and an associated set of “results,” and it learns how they connect by essentially computing lines of varying weights connecting them. Once the network has learned how to connect these training facts to the outputs, it can take any new set of inputs and predict the outcome or it can predict the best set of inputs to produce a desired outcome.

Our brains do essentially the same thing. We are exposed to “facts” and their associated outcomes every moment of every day. As these new “training sets” arrive, our biological neural network connections are physically weighted. Some become stronger, others weaker. The more often we observe a connection, the stronger that neural connection becomes. At some point it becomes so strong that it becomes undeniably obvious “common sense” to us. Unreinforced connections, like memories, become so weak they are eventually forgotten.

Note that this happens whether we know it or not and whether we want it to happen or not. We cannot NOT learn facts. We learn language as children just by overhearing it, whether we intend to learn it or not. Our neural network training does not require conscious effort and cannot be “ignored” by us. If we hear a “fact” often enough, it keeps getting weighted heavier until it eventually becomes “undeniably obvious” to us.

Pretty amazing right? It is. But here is one crucial limitation. Neither computer or biological neural networks have any intrinsic way of knowing if a training fact is valid or complete nonsense. They judge truthiness based only upon their weighting. If we tell a neural network that two plus two equals five, it will accept that as a fact and faithfully report five with complete certainty as the answer every time it is asked. Likewise, if we connect spilling salt with something bad happening to us later, that becomes a fact to our neural network of which we feel absolutely certain.

This flaw wasn’t too much of a problem during most of our evolution as we were mostly exposed to real, true facts of nature and the environment. It only becomes an issue when we are exposed to abstract symbolic “facts” which can be utter fantasy. Today, however, most of what is important to our survival are not “natural” facts that can be validated by science. They are conceptual ideas which can be repeated and reinforced in our neural networks without any physical validation. Take the idea of a god as one perfect example. We hear that god exists so often that our “proof of god” pathways strengthen to the point that we see proof everywhere and god’s existence becomes intuitively undeniable to us.

This situation is exacerbated by another related mental ability of ours… rationalization. Since a neural network can happily accommodate any “nonsense” facts, regardless of how contradictory they may be, our brains have to be very good at rationalizing away any logical discrepancies between them. If two strong network connections logically contradict each other, our brains excel and fabricating some reason, some rationale to explain how that can be. When exposed to contradictory input, we feel disoriented until we rationalize it somehow. Without that ability, we would be paralyzed and unable to function.

This ability of ours to rationalize anything is so powerful that even brain lesion patients who believe they only have half of a body will quickly rationalize away any reason you give them, any evidence you show them, that proves they are wrong. Rationalization allows us to continue to function, even when our neural networks have been trained with dramatically nonsensical facts. Further, once a neural network fact becomes strong enough, it can no longer be easily modified even by contradictory perceptions, because it filters and distorts subsequent perceptions to accommodate it. It can no longer be easily modified by even our memories as our memories are recreated in accordance with those connections every time we recreate them.

As one example to put all this together, when I worked in the Peace Corps in South Africa a group of high school principals warned me to stay indoors after dark because of the witches that roam about. I asked some questions, like have you ever personally seen a witch? No, was the answer, but many others whom we trust have told us about them. What do they look like, I asked. Well they look almost like goats with horns in the darkness. In fact, if you catch one they will transform into a goat to avoid capture.

Here you clearly see how otherwise smart people can be absolutely sure that their nonsensical “facts” and rationalizations are perfectly reasonable. What you probably don’t see is the equally nonsensical rationalizations of your own beliefs in god and souls and angels or other bizarre delusions.

So our neural networks are always being modified, regardless of how smart we are, whether we want them to or not, whether we know they are or not, and those training facts can be absolutely crazy. But our only measure of how crazy they are is our own neural network weighting which tells us that whatever are the strongest connections must be the most true. Further, our perceptions and memories are modified to remain in alignment with that programming and we can fabricate any rationalization needed to explain how our belief in even the most outlandish idea is really quite rational.

In humans early days, we could live with these inherent imperfections. They actually helped us survive. But the problems that face us today are mostly in the realm of concepts, symbols, ideas, and highly complex abstractions. There is little clear and immediate feedback in the natural world to moderate bad ideas. Therefore, the quality of our answers to those problems and challenges is entirely dependent upon the quality of our basic neural network programming.

The scientific method is a proven way to help ensure that our conclusions align with reality, but science can only be applied to empirically falsifiable questions. Science can’t help much with most of the important issues that threaten modern society like should we own guns or should Donald Trump be President. Our flawed neural networks can make some of us feel certain about such questions, but how can we be certain that our certainty is not based on bad training facts?

First, always try to surround yourself by “true and valid” training facts as much as possible. Religious beliefs, New Age ideas, fake news, and partisan rationalizations all fall under the category of “bad” training facts. Regardless of how much you know they are nonsense, if you are exposed to them you will get more and more comfortable with them. Eventually you will come around to believing them no matter how smart you think you are, it’s simply a physical process like the results of eating too much fat.

Second, the fact that exposing ourselves to nonsense is so dangerous gives us hope as well. While it’s true that deep network connections, beliefs, are difficult to change, it is a fallacy to think they cannot change. Indoctrination works, brainwashing works, marketing works. Repetition and isolation from alternative viewpoints, as practiced by Fox News, works. So we CAN change minds, no matter how deeply impervious they may seem, for the better as easily as for the worse. Education helps. Good information helps.

There is a method called Feldenkrais which can be practiced to become aware of our patterns of muscle movement, and to then strip out “bad” or “unnecessary” neural network programming to improve atheletic efficiency and performance. I maintain that our brains work in essentially the same way as the neural networks that coordinate our complex movements. As in Feldenkrais, we can slow down, examine each tiny mental step, become keenly aware of our thinking patterns, identify flaws, and correct them. If we try.

Third, rely upon the scientific method wherever you can. Science, where applicable, gives us a proven method to bypass our flawed network programming and compromised perceptions to arrive at the truth of a question.

Fourth, learn to quickly recognize fallacies of logic. This can help you to identify bad rationalizations in yourself as well as others. Recognizing flawed rationalizations can help you to identify bad neural programming. In my book Belief in Science and the Science of Belief, I discuss logical fallacies in some detail as well a going deeper into all of the ideas summarized here.

Finally, just be ever cognizant and self-aware of the fact that whatever seems obvious and intuitive to you may in fact be incorrect, inconsistent, or even simply crazy. Having humility and self-awareness of how our amazing yet deeply flawed neural networks function helps us to remain vigilant for our limitations and skeptical of our own compromised intuitions and rationalizations.

A Spoonful of Superstition

Anyone who follows my blog or has read my book, Belief in Science and the Science of Belief, well knows that I argue passionately against most forms of belief-based thinking. But I do have to admit that sometimes a teeny tiny bit of false belief is helpful.

I rarely ever steal anything. If I realize that I was given too much change at the market, I’ll invariably drive back to return it. I don’t do the right thing because I’m such a noble upstanding person. I do it because of superstition.

You see somewhere I picked up this deeply ingrained notion that if I keep that money, the universe will somehow take revenge on me. I’ll stub my toe, or get a traffic citation, or spill coffee on my term paper. In some way I’ll pay tenfold for my dishonesty.

Of course that’s silly. I know that’s silly. Something bad will happen eventually, and when it does I will falsely ascribe the cause to be my earlier bad behavior. But nevertheless, the feeling of wanting to avoid this bad karma is so powerful that it still compels me to do the right thing.

Perhaps this belief in karma is just something learned from popular culture, but I rather suspect that our species carries the hardwired seed of this idea which gets reinforced by false associations acquired during ones lifetime. It makes evolutionary sense to me that such an assumption is a helpful genetic trait. Within social structures, what goes around does in fact often come back around.

Our innate fear of karmic payback is almost certainly a component of what we perceive as our nagging conscience.

karma

Here’s the danger in this. It leads many people down the rabbit hole into religion-land. If we believe in a little auto-karma, why not an almighty god who personally ensures payback? If a little karmic fear is good, then the fear of eternal damnation is even better. If our sense of personal karma is hardwired, then so is our belief in god!

But one essential principle you learn in toxicology class is that everything that is beneficial at low doses becomes toxic at higher doses. A little superstition may have some positive benefits. But a full-blown belief in god requires a level of superstition and a perversion of logical thinking that is so extreme as to be dangerously toxic. This level of belief is so debilitating to our logical faculties that it can compromise our ability to think rationally about critical issues like climate change.

So feel free to embrace a healthy sense of karma like you do a magic show, with a full and complete understanding that it’s not real. But don’t let that indulgence lead you down the treacherously slippery slope into religion.

The Multiverse is Bigger than God

MultiverseOur gods used to be gods of specific things; the sky, the sea, war, love. Then God took over and became the god of everything. But our understanding of “everything” keeps expanding, and as it does, our fanciful notion of God has to expand along with it to remain ever beyond the limits of mere science.

The visible horizon of our observable universe is 46.5 billion light years away in any direction. That is an immense distance, and this visible sphere around us contains about 100 billion galaxies, each with perhaps 100 billion stars. Our God of everything created all that too, presumably just for us to look at.

But wait, there’s more, much more. Today we understand that our universe is almost certainly unimaginably larger than that which we can observe. It is perhaps 100 billion trillion times larger than our observable universe. That makes what we can see just the tiniest mote of dust in our greater universe. In our observable universe we can look into the sky and at least see what happened in the distant past. We can not even see out into the darkness beyond that. But since it apparently exists, believers have no choice except to inflate God once more. God presumably created all that inaccessible space beyond the horizon as well, and just for us.

It gets better. Now we are beginning to understand that God apparently created an infinite multiverse just for us as well. I first recall being fascinated by the idea of multiple universes in 1966 when Mr. Spock met Captain Kirk’s evil counterpart from an alternate universe (see here). But just as Star Trek communicators became everyday reality, the science fiction of multiple universes has become legitimate science.

There are many forms that the multiverse may take, but for now let it suffice to think of an infinite number of universes just like ours, maybe isolated in pockets of space, maybe superimposed upon each other, maybe both. Their infinity extends through both time and space. This infinite multiverse is not static. In it (if the word “in” even applies to an infinite space) universes appear, grow old, and die. Each is born with a particular set of fundamental parameters. Only a relatively tiny (but still infinite) fraction have parameters in the “Goldilocks” range that allow organized structures. In a tiny fraction of those, life is possible. The rest are stillborn or survive for a short while as unsustainable regions of chaos.

How can it get more mind-blowing? Well it is an inescapable logical conclusion is that in an infinite multiverse everything that could possibly happen must happen. For example, there must be a universe in which every possible variation of our own exists, in fact there must be an infinite number of each possible variation – infinite numbers of each of us.

Whatever form it takes, we become even more insignificant within the time-space grandeur of the multiverse. So our notion of God must once again expand dramatically to exceed even the non-existent bounds of an already infinite multiverse in order to remain the unbounded God of all things. And of course God created that infinite multiverse, so far beyond our ability to grasp let alone interact with, just for we infinitesimal humans.

I talk about god here knowing full well that it is of course completely silly to do so. I might as well talk about the how our notion of Santa Claus must expand to encompass the belief that he has to deliver Christmas presents to all children in the multiverse on one night. Yet, unfortunately we do focus our attention on our fantasy of god whenever these cosmological discussions take place.

Some “religious scholars” try desperately to keep god relevant in the face of our growing awareness by arguing that in a multiverse in which all things are possible, god must exist somewhere. In an otherwise decent article author Mark Vernon (see here), perpetuates this fallacy by repeating that since “everything is possible somewhere … it would have to conclude that God exists in some universes.

This will certainly keep getting repeated but it is simply not a correct interpretation of the science to say that in a multiverse “everything is possible.” This is a perversion of the correct formulation which is “everything possible must happen.” These are completely different ideas. Any particular universe is still governed by its own physics and there is a limit to the possible physics of any given universe. Impossible things, like gods and ghosts, can not happen in any universe.

And even if some universe had some being approaching a god, it would still not be an omnipotent god of everything and it would certainly not be our god. Therefore I am not sure how claiming that a God exists in some other universe does anything but admit that one does not exist in our own.

So what is the most rational of the possible irrational responses for someone clinging to their belief in god in the face of a multiverse? The best would be simply to claim that god created the multiverse and not even try to invoke any pseudo-scientific arguments. As you always have, just keep expanding your definition of god to supersede whatever new boundaries science reveals.

But really, adding God to the multiverse is simply adding fake infinity on top of real infinity. Like infinity plus infinity, the extra infinity is entirely superfluous and unnecessary. And what does it add to place God beyond infinity? It only replaces the insistence that something had to create the multiverse with an acceptance that nothing had to create God. It’s silly, especially given the fact that our limited concept of “before” has little relevance in an infinite multiverse.

Better yet would be to finally give in and acknowledge that the multiverse has rendered your god small and insignificant and kind of pathetic. God is like a quaint old Vaudeville act that can no longer compete with huge 3-D superhero blockbusters, and looks silly trying. Back in the day, it might have been an understandable conceit to believe that God created the Earth just for us… or even maybe the solar system. But the level of conceit required to believe that some God created the entire multiverse just for us is wildly absurd. The idea that such a God would be focused on us is insanely narcissistic.

The multiverse forces God to grow SO large, that it swells him far beyond any relevance to us or us to him.

So abandon your increasingly simplistic idea of god and find comfort, wonder, and inspiration in our incredible multiverse. You do not need to feel increasingly insignificant and worthless in this expanding multiverse. You don’t need God to give you a phony feeling of significance and meaning within it. All it takes is the flip of a mental soft-switch and you can find comfort and wonder and meaning in our amazing multiverse. It’s all just in your head after all.

I do not share the pessimism of some that we can never “see” or understand the multiverse. My working assumption is that even the greater multiverse is our cosmos, that it is knowable. If we survive Climate Change, we may eventually understand it more fully through indirect observations or through the magical lens of mathematics. Until then, if you are intrigued and stimulated by these real possibilities, I highly recommend that you read the excellent overview article by Robert Lawrence Kuhn (see here).

The Day I Met Lucy Liu

LucyLiuSo speaking of celebrity encounters in NYC, one time I was waiting for a date in front of a restaurant near my apartment in lower Manhattan, when Lucy Liu walked up and stood next to me. She was waiting for someone as well and we exchanged pleasantries.

Oh wait… that never happened. In fact, once I impulsively related this story to a group of friends. As I did, I had a growing feeling of uneasiness as if something was wrong. Then one of my friends said, “hey that’s the story I told you!” It then hit me that I was remembering her experience. I was so mortified that I tried clumsily to cover up for my humiliation.

I had had the actual experience of waiting outside that same restaurant several times to meet dates. I had had actual experiences of encountering other celebrities in NYC. I was obviously familiar with Lucy Liu from television and movies. The presence of my friend who told me that story probably triggered a temporary conflation of all of these to produce a false memory.

How can this happen? Well it happens all the time but we are not usually made aware of it in such an embarrassing fashion. You see, we do not recall events like a tape recorder. Rather our memories are like notes scribbled along the margins of our brains, associated by proximity or by little arrows. When we try to retrieve a memory, we do not hit “replay,” rather we check our scribbled, fragmented notes and try to piece the events back together as best we can. Our memories are not recordings, but are rather more like docudramas that we recreate from scraps of information each time we invoke them.

These little docudrama memories are never recreated exactly. They are full of errors and omissions. Not only are they constructed incompletely, but they are colored by our biases, fears, hopes, and needs at the time we recreate them. Even worse, each time we recall them, our scribbled notes are updated with these changes and this modified version forms the basis of our next recollection. It’s like basing your documentary of JFK on a previous documentary. Our memories are like the message in a game of telephone, changing and morphing each time we invoke them. If my friend had not been present to point this out about Lucy Liu, I may well have further integrated her story more solidly into my own memories.

As marvelous as our memories are, these things happen. Anyone, like our President, who thinks they have “one of the great memories of all time” is simply not paying close enough attention. And it’s hard to question our very memories. After all, they form the basis of who we are and all we think we know. To question our memories is to question our own competence, our own sanity, and even the very foundation of our self identity.

Yet we must be skeptical of memories, particularly our own. Most of our memories are not as falsifiable as a misremembered encounter with Lucy Liu. There is no one to call us out of most of the memories that form our experiences and define us as who we are. Most of our memory glitches aren’t exposed on a witness stand. We recall certain things about how our parents treated us but not other things. We recall a friend insulting us when it was actually a comment made on a television show. We clearly recall a pivotal experience in our life that was in fact only a recurring dream.

I noticed that when I gave lectures, I told an abbreviated story that was a composite of a number of other stories, just to illustrate a point. Before too long I could no longer recall the individual stories but only the abbreviated composite. Each time we retell a story, that replaces our old memories and our memories change and morph over time.

You can experiment on yourself to test my claims. Just start telling some story about yourself – make it somewhat plausible. I guarantee that before too long you’ll have trouble recalling that you made this up. Before much longer, and you’ll become absolutely certain it is completely true. You remember it clearly after all, so it must be. That’s how your neural network works. You cut you bleed. You repeat experiences through stories or dreams or whatever, and they become memories. They become your self.

Even more important than recalling events is the problem of remembering feelings. Our recollection of feelings is extremely malleable. We can quickly grow to hate a dear friend or spouse largely because we retell stories over and over again that gradually deteriorate into a “powerful, unforgettable recollection” of how terribly we have been treated in the past. If we are disposed to think badly of someone, all our memories will be colored by that lens. Perhaps, a good indicator of the person we are now is the tone we impart upon the docudramas we recreate to recall events and how those events made us feel.

In fact, studies have suggested that having a “great memory” makes you less happy. People with so-called great memories tend to never let go of any slight or insult. They only get angrier and more unhappy as time goes on. I submit that their memories are not actually that great and that they likely have a tendency to create a more unforgivably offensive version of their memory each time they recall it.

Lucy Liu reminds me that a little humility when it comes to the fallibility of our own memories is a good thing. Maybe it would make Donald Trump a less angry and spiteful person if he were not quite so confident that he has “one of the great memories of all time.” Or, maybe angry and spiteful people just cannot help but create memories that reinforce their anger and spitefulness.

 

Data Management Primer

data_analysisI spent the better part of 40 years working with data. As a scientific researcher and then as a software developer, my work was all about acquiring, managing, analyzing, and reporting data. That experience taught me lots of lessons. I shared many of those lessons by teaching database design and development at the college level and by writing books on the topic.

Chances are that as a professional in the information age, you do a lot of work with data as well. Here’s the most important thing to know right off… this doesn’t come naturally. Collecting and storing data properly is deceptively difficult. I know because as a software consultant I was called in to fix innumerable systems that were failing catastrophically because they were grown organically by clever, smart people using spreadsheets with complete confidence they were doing a great job.

Even good data gets ruined if it is badly stored.  This applies to small projects as much as large ones. Poorly stored data becomes the “garbage in” for your subsequent analyses and reports. Therefore I thought I would share a brief summary of some of the most important “do’s and don’ts” when I comes to collecting and storing your data. I won’t try to explain or justify all of these rules, but trust me, if you follow them your work will be far more efficient and effective:

  1.  Always create a key field. This field should uniquely identify each record. I recommend you identify each record with a simple incrementing number, one that has no real-word meaning and that you never report to external users of the data.
  2. Don’t create derived fields. It is generally not good practice to store fields of data that are derived or calculated from other fields. It is better to compute these when needed to ensure the values are current.
  3. Make sure every column or field of data is atomic. That is, each field value should contain only one irreducible piece of information. You never want to put say, two phone numbers in one field. Store only one piece of information per field by creating separate columns if necessary.
  4. Think ahead carefully about how atomic a field needs to be.  Will I need to break up phone numbers into separate parts? You have to consider how the data will likely be used and the best choice is still not always obvious. Do you ever need to know the street name in an address? If so, you might consider storing the house number and street in separate fields as this is extremely difficult to parse. Will you need the month from a date? You probably still don’t need to store that separately since it is extremely easy to extract months from dates when needed.
  5. Avoid series of columns. You don’t want Phone1, Phone2, and Phone3 fields. Better to have a separate “phone numbers” table with a record for each phone number along with the type of phone linked through a key field. A relational table like this is much more flexible, efficient, maintainable, and expandable.
  6. Don’t stick data where it doesn’t belong. When you don’t have a column, it is tempting to just jam some values into a column where they do not really belong. For example, you have a pregnancy field that does not apply to men. So why not stick prostate indicator into this column in records for males? This is a major no no. Every value in a column should describe only one attribute.
  7. Don’t append or prepend. Always avoid appending or prepending data values to add further information. For example, adding “fax” or “mobile” after phone numbers. Instead add a separate phone type column or better yet create a separate relational table to store these values.
  8. Don’t vary record types in a given table. You can have a column that indicates whether the record is a Parent or Child, but you should not then change the meaning of subsequent fields depending on whether the record belongs to a parent or a child. If parents and children require different information, create separate tables.
  9. Never use “special” values. Never put in a reserved value like “NONE” or “1/1/1999” “to indicate some special condition. This is a very bad practice that inevitably results in errors, hair-pulling, and tooth-grinding.
  10. Give your columns clear and meaningful names. Don’t use cryptic names. If the column contains first names, simply call the column “First Name.” This makes it completely unambiguous, reduces errors in analysis, and makes reporting clear, consistent, and easy
  11.  Format your data consistently. For example, store SSN with hyphens or without as you prefer, but don’t mix these. With hyphens is preferable since it is best to store data so that it does not require formatting upon reporting.
  12. Use Null and Empty values correctly. -Assuming your data storage system supports these, use them correctly. Null means that value was never entered. Empty means it was entered, and it was explicitly entered as empty.
  13. Allow Null and Empty responses on your forms. Forcing users to put in an answer they don’t know yet just to save the form only opens the door to garbage data. If you force an answer too early, you may be forcing the user to make something up just to move on, and later there is no way to spot this as “fake” data. Better to wait until the last possible point prior to finalization before verifying that all data entry is complete.
  14. Don’t allow bad values to be entered. Provide dropdown or selection lists wherever possible rather than text data entry. Text (freeform) data values are usually garbage data values that are very problematic to search, analyze, and report.
  15. Don’t duplicate any information. No information should be repeated in rows. For example, you should have one parent table with customers and another child table with orders records for each customer. You should not have one “flat” table which repeats the same customer information on every order record for that customer. This is one key difference between simple “flat table” data storage and a normalized relational database.
  16. Break the rules when needed. The only thing worse than not following these rules is mindlessly following them. Break these rules only when it makes good sense to break them. You can’t make that assessment if you do not understand them, and the reasons for them, intimately.
  17. Think long-term. Don’t assume you’ll never need your data again or that you will be the only one to ever look at it. You may know how you violated these rules, but others might need to understand this data in the future and they will not. Following these rules will ensure that this valuable data you collected is not wasted just because you could not anticipate how they might be used in the future.

Some of these rules simply cannot be achieved using a single (denormalized) table like you typically create with Excel. They require multiple tables following a normalized design structure of parent and child tables.

If you don’t see any way you can avoid breaking these rules, then your data storage requirements probably exceed the limitations of a simple table. If that is the case, the fact that you cannot implement these rules should alert you that you need to consult a data management professional to produce an efficient relational database schema using SQL Server or some other professional database.