Category Archives: Science

Don’t Believe your Eyes

1 Reply

eyesToday I wanted to talk about perceptions. Not our feelings, but what we actually see, feel, smell, hear, and taste. That is, the “objective” inputs that drive our feelings. Should we really “only believe our eyes?”

I think not.

In my book (see here) I talk about how we should be skeptical of our own memories and perceptions. Our memories are not recordings. They are docudrama recreations drawing upon various stock footage to put together a satisfying re-imagining. We remember going to the beach as a child. But in “recalling” details of that experience, we draw upon fragments from various sources to fill it in. The “slant” of that recreation is strongly dependent upon our current attitudes and biases. Our re-imagined, and often very distorted memory then reinforces what we believe to be a “vivid” recollection next time we recall it. Over time our “clear” memory can drift farther and farther from reality like a memory version of the “phone game.”

I contend that our brains work similarly with regard to our senses. We don’t see what we think we see. Our perceptions are filtered through our complex neural networks. It is a matched, filtered, processed, censored, and often highly biased version that we actually see, hear, or feel.

We know that our subconscious both filters out much of the information it receives, and adds in additional information as needed to create a sensible perception. I always favor a neural network model of brain function. As it relates to perception, our neural network receives a set of sensory data. It matches that data against known patterns and picks the closest match. It then presents our consciousness with a picture – not of the original data – but of that best-fit match. It leaves out “extraneous” information and may add in missing information to complete that expected picture. That is, we do not actually see, hear, smell, or taste a thing directly. We see, hear, smell, or taste a satisfying recreation that our network presents to us.

This should not be controversial, because we experience it all the time. Based on sparse information, we “see” fine detail in a low resolution computer icon that objectively is not there. We fail to see the gorilla inserted into the background because it is out of place. We are certain we see a witch or a vase in a silhouette, depending on our bias or our expectations at that moment.

But though this should be evident, we still do not take this imprecision seriously enough in evaluating the objectivity of our own memories or perceptions. We still mostly put near-absolute faith in our memories, and are generally even more certain of our perceptions. We believe that what we perceive is absolutely objective. Clearly, it is not.

In essence, what we believe we objectively recall, see, hear, or touch is not the thing itself, but a massaged recreation of our neural network match. The version we perceive can often be wrong in very important ways. Our perceptions are only as reliable as our neural networks. And some neural networks can be more compromised than others. We can recall or even perceive radically crazy things if our neural network has been trained to do so. I campaign against belief-based thinking of all sort because it seriously compromises these critical neural networks in crazy ways.

Even more unrecognized are the ways that this phenomenon is largely ignored as it impacts scientific research. Scientists often give far too much credence to reports of perceptions, often in extremely subtle ways.

As a simple illustration, consider how we often mock wine connoisseurs who claim to taste differences in wines but cannot pick these out in blinded studies. However, consider the confounding impact of their (and our) neural networks in even this simple case. When experiencing a wine, all the associated data is fed into the drinker’s neural network. It makes a match and then presents that match to the consciousness. Therefore, if the network does not “see” one critical factor, say color, it matches to white, not red, and presents and entirely different taste pattern the the drinker, ignoring some “extraneous” flavors and adding some other “missing” ones.

These same kinds of neural network matching errors can, and I have to assume often do, confound even more rigorous scientific studies. And they are further confounded by the fact that these mismatches are typically temporary. With every new set of data, our neural networks adjust themselves, the weightings change, to yield different results. The effect of a drug or placebo, for example, may change over time. If scientists see this, they typically look exclusively for other physiological causes. But it may be a neural network correction.

That is why I always admonish my readers to stick with inputs that will strengthen your neural networks toward sound objectivity rather than allow them to be weighted toward the rationalization of, and perception of, beliefs and nonsense. But since none of us can ever have perfect networks, or even know how accurate ours performs in any given match, we all need a healthy amount of skepticism, even with regard to our own memories and perceptions.

I further urge scientists to at least consider the impact of neural network pre-processing on your studies, and to develop methodologies to explicitly detect and correct for such biases.

 

Humans are Inexplicable

1 Reply

brainWhether it be in science or business or politics or popular culture, we expend an inordinate amount of time and effort trying to figure out why people do whatever people are doing. We seem to have more analysts than actors, all desperately trying to explain what motivates people, either by asking them directly or by making inferences about them. For the most part, this is not merely a colossal waste of time and effort and money in itself, but it stimulates even greater wastes of time and effort and money chasing wildly incomplete or erroneous conclusions about why we do what we do.

Asking people why they did what they did, or why they are doing what they are doing, or why they are going to do what they are going to do, generally yields useless and misleading information. It is not clear that people actually have distinct reasons they can recognize let alone articulate. It is quite likely in fact that most of the decisions we make are made unconsciously based upon a myriad of complex neural network associations. These associations need not be rational. These connections don’t need to be internally consistent to each other or related to the actual outcome in any way. But in our post-rationalizations and post-analyses we impose some logic to our decisions to make them feel sensible. Therefore, the reasons we come up with are almost completely made-up at every level to sound rational or at least sane to ourselves and to those we are communicating to.

The truth is, we can’t usually hope to understand our own incredibly complex neural networks, let alone the neural networks of others. Yes, sometimes we can identify a strong neural network association driving a behavior, but most determinative associations are far too diffuse across a huge number of seemingly unrelated associations.

The situation gets infinitely worse when we are trying to analyze and explain group behaviors. Most of our shared group behaviors emerge from the weak-interactions between all of our individual neural networks. The complexity of these interactions is virtually unfathomable. The challenge of understanding why a group does what it does collectively, let alone figuring out how to influence their behavior, is fantastic.

If you ask a bird why it is flying in a complex swirling pattern along with a million other birds, it will probably give you some reason, like “we are looking for food,” but in fact it is probably largely unaware that it is even flying in any particular pattern at all.

So why point all this out? Do we give up? Does this imply that a rational civilization is impossible, that all introspection or external analysis is folly?

Quite the contrary, we must continue to struggle to understand ourselves and truly appreciating our complexity is part of that effort. To do so we must abandon the constraints of logic that we impose upon our individual and group rationalizations and appreciate that we are driven by neural networks that are susceptible to all manner of illogical programming. We must take any self-reporting with the same skepticism we would to the statement “I am perfectly sane.” We should be careful of imposing our own flawed rationality upon the flawed rationality of others. Analysts should not assume undue rationality in explaining behaviors. And finally, we must appreciate that group behaviors can have little or no apparent relationship to any of the wants, needs, or expressed opinions of those individuals within that group.

In advanced AI neural networks, we humans cannot hope to understand why the computer has made a decision. Its decision is based upon far too many subtle factors for humans to recognize or articulate. But if all of the facts programmed in to the computer are accurate, we can probably trust the judgement of the computer.

Similarly with humans, it may be that our naive approach of asking or inferring reasons for feelings and behaviors and then trying to respond to each of those rationales is incredibly ineffective. It may be that the only thing that would truly improve individual and thus emergent thinking are more sanely programmed neural networks, ones that are not fundamentally flawed so as to comfortably rationalize religious and other specious thinking at the most basic level (see here). We must focus on basic fact-based thinking in our educational system and in our culture on the assumption that more logically and factually-trained human neural networks will yield more rational and effective individual and emergent behaviors.

 

Religion in Public Schools

1 Reply

The teaching of religion in public schools is a topic that stimulates a great deal of honest debate on all sides of the issue. Should religion be taught at all? And if so, what religions? Even well-meaning atheists might feel that religion should be taught, as long as all religions – and atheistic perspectives as well – are taught equally and fairly without bias.

That sounds laudable and enlightened in theory. However, many plans that sound great in theory inevitably turn out to be disastrous when put into practice. Teaching religion in public schools is one such example.

I have personal experience with this. While serving in the Peace Corps in South Africa, I worked for their Department of Education. The South African Constitution requires that all religions be treated equally. In order to comply with the spirit of their Constitution, the Department of Education has adopted a policy that all religions should be taught fairly and equally in the public schools.

Sounds great right? The trouble is that teachers, particularly rural teachers, do not know all religions and do not care to know all religions – let alone teach them fairly. At the point where lofty policies touch the students. all that this accomplishes is to give teachers cover to preach and proselytize their own religious views in the classroom and to misrepresent and disparage all other religions – and atheism is demonized most of all.

The problem of state sanctioned religious instruction is not merely a matter of the recruiting and training and monitoring of teachers. False even-handedness spills over into teaching materials as well. Science texts typically enumerate a long list of native creation myths as legitimate. In at least one science text, after describing the monkey myth, and the milk myth, and many others, it concluded with what was almost an obligatory footnote that said “and some scientists believe that the world was created by natural means and human beings evolved.”

This sort of false balance, not unlike giving equal deference to climate change deniers, is an almost inevitable consequence of a misguided and ill-fated attempt to be fair and inclusive with regard to the teaching of religion.

I came away from my experience in South Africa more convinced than ever that our American system of simply keeping religion out of our public schools is on balance the best, most practical system of fairness. There is no shortage of alternate venues where people can preach and teach religion as much as they wish. Therefore, there is no compelling need being met by including religion in public schools, that warrants the certain risk of abuse and unintended consequences.

Assiduously keeping religion out of our public schools is in fact the more fair, the more enlightened, and the more realistic policy position.

Our Amazing Yet Deeply Flawed Neural Networks

Leave a reply

NeuralNetwork

Back in the 1980’s when I did early work applying Neural Network technology to paint formulation chemistry, that experience gave me fascinating insights into how our brains operate. A computer neural network is a mathematically complex program that does a simple thing. It takes a set of training “facts” and an associated set of “results,” and it learns how they connect by essentially computing lines of varying weights connecting them. Once the network has learned how to connect these training facts to the outputs, it can take any new set of inputs and predict the outcome or it can predict the best set of inputs to produce a desired outcome.

Our brains do essentially the same thing. We are exposed to “facts” and their associated outcomes every moment of every day. As these new “training sets” arrive, our biological neural network connections are physically weighted. Some become stronger, others weaker. The more often we observe a connection, the stronger that neural connection becomes. At some point it becomes so strong that it becomes undeniably obvious “common sense” to us. Unreinforced connections, like memories, become so weak they are eventually forgotten.

Note that this happens whether we know it or not and whether we want it to happen or not. We cannot NOT learn facts. We learn language as children just by overhearing it, whether we intend to learn it or not. Our neural network training does not require conscious effort and cannot be “ignored” by us. If we hear a “fact” often enough, it keeps getting weighted heavier until it eventually becomes “undeniably obvious” to us.

Pretty amazing right? It is. But here is one crucial limitation. Neither computer or biological neural networks have any intrinsic way of knowing if a training fact is valid or complete nonsense. They judge truthiness based only upon their weighting. If we tell a neural network that two plus two equals five, it will accept that as a fact and faithfully report five with complete certainty as the answer every time it is asked. Likewise, if we connect spilling salt with something bad happening to us later, that becomes a fact to our neural network of which we feel absolutely certain.

This flaw wasn’t too much of a problem during most of our evolution as we were mostly exposed to real, true facts of nature and the environment. It only becomes an issue when we are exposed to abstract symbolic “facts” which can be utter fantasy. Today, however, most of what is important to our survival are not “natural” facts that can be validated by science. They are conceptual ideas which can be repeated and reinforced in our neural networks without any physical validation. Take the idea of a god as one perfect example. We hear that god exists so often that our “proof of god” pathways strengthen to the point that we see proof everywhere and god’s existence becomes intuitively undeniable to us.

This situation is exacerbated by another related mental ability of ours… rationalization. Since a neural network can happily accommodate any “nonsense” facts, regardless of how contradictory they may be, our brains have to be very good at rationalizing away any logical discrepancies between them. If two strong network connections logically contradict each other, our brains excel and fabricating some reason, some rationale to explain how that can be. When exposed to contradictory input, we feel disoriented until we rationalize it somehow. Without that ability, we would be paralyzed and unable to function.

This ability of ours to rationalize anything is so powerful that even brain lesion patients who believe they only have half of a body will quickly rationalize away any reason you give them, any evidence you show them, that proves they are wrong. Rationalization allows us to continue to function, even when our neural networks have been trained with dramatically nonsensical facts. Further, once a neural network fact becomes strong enough, it can no longer be easily modified even by contradictory perceptions, because it filters and distorts subsequent perceptions to accommodate it. It can no longer be easily modified by even our memories as our memories are recreated in accordance with those connections every time we recreate them.

As one example to put all this together, when I worked in the Peace Corps in South Africa a group of high school principals warned me to stay indoors after dark because of the witches that roam about. I asked some questions, like have you ever personally seen a witch? No, was the answer, but many others whom we trust have told us about them. What do they look like, I asked. Well they look almost like goats with horns in the darkness. In fact, if you catch one they will transform into a goat to avoid capture.

Here you clearly see how otherwise smart people can be absolutely sure that their nonsensical “facts” and rationalizations are perfectly reasonable. What you probably don’t see is the equally nonsensical rationalizations of your own beliefs in god and souls and angels or other bizarre delusions.

So our neural networks are always being modified, regardless of how smart we are, whether we want them to or not, whether we know they are or not, and those training facts can be absolutely crazy. But our only measure of how crazy they are is our own neural network weighting which tells us that whatever are the strongest connections must be the most true. Further, our perceptions and memories are modified to remain in alignment with that programming and we can fabricate any rationalization needed to explain how our belief in even the most outlandish idea is really quite rational.

In humans early days, we could live with these inherent imperfections. They actually helped us survive. But the problems that face us today are mostly in the realm of concepts, symbols, ideas, and highly complex abstractions. There is little clear and immediate feedback in the natural world to moderate bad ideas. Therefore, the quality of our answers to those problems and challenges is entirely dependent upon the quality of our basic neural network programming.

The scientific method is a proven way to help ensure that our conclusions align with reality, but science can only be applied to empirically falsifiable questions. Science can’t help much with most of the important issues that threaten modern society like should we own guns or should Donald Trump be President. Our flawed neural networks can make some of us feel certain about such questions, but how can we be certain that our certainty is not based on bad training facts?

First, always try to surround yourself by “true and valid” training facts as much as possible. Religious beliefs, New Age ideas, fake news, and partisan rationalizations all fall under the category of “bad” training facts. Regardless of how much you know they are nonsense, if you are exposed to them you will get more and more comfortable with them. Eventually you will come around to believing them no matter how smart you think you are, it’s simply a physical process like the results of eating too much fat.

Second, the fact that exposing ourselves to nonsense is so dangerous gives us hope as well. While it’s true that deep network connections, beliefs, are difficult to change, it is a fallacy to think they cannot change. Indoctrination works, brainwashing works, marketing works. Repetition and isolation from alternative viewpoints, as practiced by Fox News, works. So we CAN change minds, no matter how deeply impervious they may seem, for the better as easily as for the worse. Education helps. Good information helps.

There is a method called Feldenkrais which can be practiced to become aware of our patterns of muscle movement, and to then strip out “bad” or “unnecessary” neural network programming to improve atheletic efficiency and performance. I maintain that our brains work in essentially the same way as the neural networks that coordinate our complex movements. As in Feldenkrais, we can slow down, examine each tiny mental step, become keenly aware of our thinking patterns, identify flaws, and correct them. If we try.

Third, rely upon the scientific method wherever you can. Science, where applicable, gives us a proven method to bypass our flawed network programming and compromised perceptions to arrive at the truth of a question.

Fourth, learn to quickly recognize fallacies of logic. This can help you to identify bad rationalizations in yourself as well as others. Recognizing flawed rationalizations can help you to identify bad neural programming. In my book Belief in Science and the Science of Belief, I discuss logical fallacies in some detail as well a going deeper into all of the ideas summarized here.

Finally, just be ever cognizant and self-aware of the fact that whatever seems obvious and intuitive to you may in fact be incorrect, inconsistent, or even simply crazy. Having humility and self-awareness of how our amazing yet deeply flawed neural networks function helps us to remain vigilant for our limitations and skeptical of our own compromised intuitions and rationalizations.

Technology Empowers Our Humanity

Leave a reply

CustomerSupportNot that may years ago, read/write CD/ROM drives were essential and a good one was quite expensive. I once paid top dollar to get a top rated drive from Toshiba. It never worked. I called Toshiba dozens of times over 6 months trying to get it working. It would take an hour to get past hold, read off serial numbers and customer info, fax in receipts, explain the problem all over again, to get transferred and repeat it all, to get disconnected, go through it all yet again, only to be told to clean the drive, to call Microsoft, to contact Intel, to reinstall Windows, to buy higher quality disks, to change bios settings, or buy a new connection cable.

In the end, it turned out that this was a known issue with the drive, but Toshiba had a policy not to admit to any such issues. Instead, they intentionally made me jump onerous technical support hurdles and run off on expensive and time-consuming wild goose chases for six months before they finally admitted as much. Most people gave up well before that, but I was on a mission. Nevertheless, in the end I tossed the drive in the garbage.

Everyone has their customer support horror stories. Not that long ago, such infuriating experiences were the norm, not the exception. I had many similar experiences with Sony in particular and resolved never to buy anything from them ever again.

But today customer support has transformed dramatically. Today, wonderful customer support is the norm, not the exception.

AT&T exemplifies this welcome new normal for customer service. The hotspot on my mobile phone quit working. Although I knew it was not an issue with AT&T because it worked on my wife’s phone, I went to their site, hit chat, immediately got a wonderful representative named Stephanie who happily helped me reset my phone, 5 minutes later my hotspot was working!

That’s great customer service. And it’s not just huge companies that are putting the service back in customer service. My garage door light started blinking in a regular pattern as if indicating some error. I called Guardian Garage Doors and immediately got a wonderful guy on the phone. He heard my issue and asked me to text him a video. I did so and after a short hold said their engineers didn’t know what the problem was but wanted me to send it in so they could diagnose it. He offered to rush out a replacement. But minutes later he called back and suggested I try replacing my LED bulb. I did so even though it seemed silly, LED’s don’t do that. But apparently they do. That fixed it!

This is nothing remotely like the bad old days of Toshiba and Sony era customer “support.” The kind of great customer support we often see today is greatly facilitated by technology. It is enabled by the Internet, by chat technology, by searchable knowledge bases, by intelligent call routing systems, and by interconnected global workforces.

But while these technologies are incredibly empowering, real people and attitudes are still essential to great customer support. Technology doesn’t make representatives so pleasantly informal yet professional in demeanor. Technology doesn’t ensure that customer service departments are staffed to connect quickly and to stay on as long as it takes to resolve an issue. It takes sensible management to not interrogate you to prove your identity, ownership, and warranty. It is an explicit choice to authorize representatives to own issues even if they are not directly responsible. And it is their conscious decision to admit to issues candidly rather than reflexively conceal and deny them beyond all rationality.

So, while I often bash private sector corporations, I must give credit where credit is due. Some things do get better. Customer service stands in direct contradiction to widespread fears of a cold and impersonal technology-dominated future. It shows us that technology, properly implemented, can make our lives and our interactions not only more efficient and satisfying, but at the same time more friendly, more personal, more sensible, and yes, more human as well.

The Multiverse is Bigger than God

1 Reply

MultiverseOur gods used to be gods of specific things; the sky, the sea, war, love. Then God took over and became the god of everything. But our understanding of “everything” keeps expanding, and as it does, our fanciful notion of God has to expand along with it to remain ever beyond the limits of mere science.

The visible horizon of our observable universe is 46.5 billion light years away in any direction. That is an immense distance, and this visible sphere around us contains about 100 billion galaxies, each with perhaps 100 billion stars. Our God of everything created all that too, presumably just for us to look at.

But wait, there’s more, much more. Today we understand that our universe is almost certainly unimaginably larger than that which we can observe. It is perhaps 100 billion trillion times larger than our observable universe. That makes what we can see just the tiniest mote of dust in our greater universe. In our observable universe we can look into the sky and at least see what happened in the distant past. We can not even see out into the darkness beyond that. But since it apparently exists, believers have no choice except to inflate God once more. God presumably created all that inaccessible space beyond the horizon as well, and just for us.

It gets better. Now we are beginning to understand that God apparently created an infinite multiverse just for us as well. I first recall being fascinated by the idea of multiple universes in 1966 when Mr. Spock met Captain Kirk’s evil counterpart from an alternate universe (see here). But just as Star Trek communicators became everyday reality, the science fiction of multiple universes has become legitimate science.

There are many forms that the multiverse may take, but for now let it suffice to think of an infinite number of universes just like ours, maybe isolated in pockets of space, maybe superimposed upon each other, maybe both. Their infinity extends through both time and space. This infinite multiverse is not static. In it (if the word “in” even applies to an infinite space) universes appear, grow old, and die. Each is born with a particular set of fundamental parameters. Only a relatively tiny (but still infinite) fraction have parameters in the “Goldilocks” range that allow organized structures. In a tiny fraction of those, life is possible. The rest are stillborn or survive for a short while as unsustainable regions of chaos.

How can it get more mind-blowing? Well it is an inescapable logical conclusion is that in an infinite multiverse everything that could possibly happen must happen. For example, there must be a universe in which every possible variation of our own exists, in fact there must be an infinite number of each possible variation – infinite numbers of each of us.

Whatever form it takes, we become even more insignificant within the time-space grandeur of the multiverse. So our notion of God must once again expand dramatically to exceed even the non-existent bounds of an already infinite multiverse in order to remain the unbounded God of all things. And of course God created that infinite multiverse, so far beyond our ability to grasp let alone interact with, just for we infinitesimal humans.

I talk about god here knowing full well that it is of course completely silly to do so. I might as well talk about the how our notion of Santa Claus must expand to encompass the belief that he has to deliver Christmas presents to all children in the multiverse on one night. Yet, unfortunately we do focus our attention on our fantasy of god whenever these cosmological discussions take place.

Some “religious scholars” try desperately to keep god relevant in the face of our growing awareness by arguing that in a multiverse in which all things are possible, god must exist somewhere. In an otherwise decent article author Mark Vernon (see here), perpetuates this fallacy by repeating that since “everything is possible somewhere … it would have to conclude that God exists in some universes.”

This will certainly keep getting repeated but it is simply not a correct interpretation of the science to say that in a multiverse “everything is possible.” This is a perversion of the correct formulation which is “everything possible must happen.” These are completely different ideas. Any particular universe is still governed by its own physics and there is a limit to the possible physics of any given universe. Impossible things, like gods and ghosts, can not happen in any universe.

And even if some universe had some being approaching a god, it would still not be an omnipotent god of everything and it would certainly not be our god. Therefore I am not sure how claiming that a God exists in some other universe does anything but admit that one does not exist in our own.

So what is the most rational of the possible irrational responses for someone clinging to their belief in god in the face of a multiverse? The best would be simply to claim that god created the multiverse and not even try to invoke any pseudo-scientific arguments. As you always have, just keep expanding your definition of god to supersede whatever new boundaries science reveals.

But really, adding God to the multiverse is simply adding fake infinity on top of real infinity. Like infinity plus infinity, the extra infinity is entirely superfluous and unnecessary. And what does it add to place God beyond infinity? It only replaces the insistence that something had to create the multiverse with an acceptance that nothing had to create God. It’s silly, especially given the fact that our limited concept of “before” has little relevance in an infinite multiverse.

Better yet would be to finally give in and acknowledge that the multiverse has rendered your god small and insignificant and kind of pathetic. God is like a quaint old Vaudeville act that can no longer compete with huge 3-D superhero blockbusters, and looks silly trying. Back in the day, it might have been an understandable conceit to believe that God created the Earth just for us… or even maybe the solar system. But the level of conceit required to believe that some God created the entire multiverse just for us is wildly absurd. The idea that such a God would be focused on us is insanely narcissistic.

The multiverse forces God to grow SO large, that it swells him far beyond any relevance to us or us to him.

So abandon your increasingly simplistic idea of god and find comfort, wonder, and inspiration in our incredible multiverse. You do not need to feel increasingly insignificant and worthless in this expanding multiverse. You don’t need God to give you a phony feeling of significance and meaning within it. All it takes is the flip of a mental soft-switch and you can find comfort and wonder and meaning in our amazing multiverse. It’s all just in your head after all.

I do not share the pessimism of some that we can never “see” or understand the multiverse. My working assumption is that even the greater multiverse is our cosmos, that it is knowable. If we survive Climate Change, we may eventually understand it more fully through indirect observations or through the magical lens of mathematics. Until then, if you are intrigued and stimulated by these real possibilities, I highly recommend that you read the excellent overview article by Robert Lawrence Kuhn (see here).

Data Management Primer

Leave a reply

data_analysisI spent the better part of 40 years working with data. As a scientific researcher and then as a software developer, my work was all about acquiring, managing, analyzing, and reporting data. That experience taught me lots of lessons. I shared many of those lessons by teaching database design and development at the college level and by writing books on the topic.

Chances are that as a professional in the information age, you do a lot of work with data as well. Here’s the most important thing to know right off… this doesn’t come naturally. Collecting and storing data properly is deceptively difficult. I know because as a software consultant I was called in to fix innumerable systems that were failing catastrophically because they were grown organically by clever, smart people using spreadsheets with complete confidence they were doing a great job.

Even good data gets ruined if it is badly stored.  This applies to small projects as much as large ones. Poorly stored data becomes the “garbage in” for your subsequent analyses and reports. Therefore I thought I would share a brief summary of some of the most important “do’s and don’ts” when I comes to collecting and storing your data. I won’t try to explain or justify all of these rules, but trust me, if you follow them your work will be far more efficient and effective:

  1.  Always create a key field. This field should uniquely identify each record. I recommend you identify each record with a simple incrementing number, one that has no real-word meaning and that you never report to external users of the data.
  2. Don’t create derived fields. It is generally not good practice to store fields of data that are derived or calculated from other fields. It is better to compute these when needed to ensure the values are current.
  3. Make sure every column or field of data is atomic. That is, each field value should contain only one irreducible piece of information. You never want to put say, two phone numbers in one field. Store only one piece of information per field by creating separate columns if necessary.
  4. Think ahead carefully about how atomic a field needs to be.  Will I need to break up phone numbers into separate parts? You have to consider how the data will likely be used and the best choice is still not always obvious. Do you ever need to know the street name in an address? If so, you might consider storing the house number and street in separate fields as this is extremely difficult to parse. Will you need the month from a date? You probably still don’t need to store that separately since it is extremely easy to extract months from dates when needed.
  5. Avoid series of columns. You don’t want Phone1, Phone2, and Phone3 fields. Better to have a separate “phone numbers” table with a record for each phone number along with the type of phone linked through a key field. A relational table like this is much more flexible, efficient, maintainable, and expandable.
  6. Don’t stick data where it doesn’t belong. When you don’t have a column, it is tempting to just jam some values into a column where they do not really belong. For example, you have a pregnancy field that does not apply to men. So why not stick prostate indicator into this column in records for males? This is a major no no. Every value in a column should describe only one attribute.
  7. Don’t append or prepend. Always avoid appending or prepending data values to add further information. For example, adding “fax” or “mobile” after phone numbers. Instead add a separate phone type column or better yet create a separate relational table to store these values.
  8. Don’t vary record types in a given table. You can have a column that indicates whether the record is a Parent or Child, but you should not then change the meaning of subsequent fields depending on whether the record belongs to a parent or a child. If parents and children require different information, create separate tables.
  9. Never use “special” values. Never put in a reserved value like “NONE” or “1/1/1999” “to indicate some special condition. This is a very bad practice that inevitably results in errors, hair-pulling, and tooth-grinding.
  10. Give your columns clear and meaningful names. Don’t use cryptic names. If the column contains first names, simply call the column “First Name.” This makes it completely unambiguous, reduces errors in analysis, and makes reporting clear, consistent, and easy
  11.  Format your data consistently. For example, store SSN with hyphens or without as you prefer, but don’t mix these. With hyphens is preferable since it is best to store data so that it does not require formatting upon reporting.
  12. Use Null and Empty values correctly. -Assuming your data storage system supports these, use them correctly. Null means that value was never entered. Empty means it was entered, and it was explicitly entered as empty.
  13. Allow Null and Empty responses on your forms. Forcing users to put in an answer they don’t know yet just to save the form only opens the door to garbage data. If you force an answer too early, you may be forcing the user to make something up just to move on, and later there is no way to spot this as “fake” data. Better to wait until the last possible point prior to finalization before verifying that all data entry is complete.
  14. Don’t allow bad values to be entered. Provide dropdown or selection lists wherever possible rather than text data entry. Text (freeform) data values are usually garbage data values that are very problematic to search, analyze, and report.
  15. Don’t duplicate any information. No information should be repeated in rows. For example, you should have one parent table with customers and another child table with orders records for each customer. You should not have one “flat” table which repeats the same customer information on every order record for that customer. This is one key difference between simple “flat table” data storage and a normalized relational database.
  16. Break the rules when needed. The only thing worse than not following these rules is mindlessly following them. Break these rules only when it makes good sense to break them. You can’t make that assessment if you do not understand them, and the reasons for them, intimately.
  17. Think long-term. Don’t assume you’ll never need your data again or that you will be the only one to ever look at it. You may know how you violated these rules, but others might need to understand this data in the future and they will not. Following these rules will ensure that this valuable data you collected is not wasted just because you could not anticipate how they might be used in the future.

Some of these rules simply cannot be achieved using a single (denormalized) table like you typically create with Excel. They require multiple tables following a normalized design structure of parent and child tables.

If you don’t see any way you can avoid breaking these rules, then your data storage requirements probably exceed the limitations of a simple table. If that is the case, the fact that you cannot implement these rules should alert you that you need to consult a data management professional to produce an efficient relational database schema using SQL Server or some other professional database.

Scientific Models

Leave a reply

I recently attended a book club discussion on The Meme Machine by Susan Blackmore (see here).  In it, Blackmore puts forth a thesis of “memetic evolution” to describe how our minds work. In fact, her assertion is that our minds can only be understood in terms of memetic selection. Although that seems to be a wildly exaggerated claim, the scientific model she proposes is both stimulating and promising.

But memetic evolution is not the topic of this article. I only cite it as one example of the kind of topic that  many non-scientists and even some scientists have great difficulty discussing fairly. Often in discussing such topics, a great many unfounded criticisms are lodged, and these quite often flow from an inadequate understanding and appreciation of scientific models.

This is understandable. Unless you are a trained, experienced, and particularly thoughtful scientist, you probably have had inadequate background to fully appreciate the concept of a scientific model. In fact, if you look up the word model in most dictionaries, the scientific usage of the term is typically not even mentioned. No wonder many people have a very limited if not completely mistaken appreciation of what a scientific model is. A scientific model is not analogous to a plastic model kit that is intended to look just like the real race car in every detail. It is not at all like a fashion model, intended to present something in an attractive manner. Nor is it like an aspirational model to be put forth as a goal to emulate and strive toward.

No, a scientific model is a working system that does not need to actually “look like” the real system it describes in any conventional way. The important characteristic of a scientific model is that it behave like the real system it describes. How accurately a scientific model reflects the real system it models is measured by how well it explains observed behaviors of the real system and is able to predict future behaviors of the real system.

For example, in 1913 Ernest Rutherford and Niels Bohr put forth the atomic model of matter that we are all familiar with – a nucleus of protons and neutrons orbited by electrons. This was a highly successful model because it described a huge number of observed characteristics and behaviors of matter, allowed us to gain great understanding of matter, and most importantly allowed us to predict as yet unobserved traits of matter.

But in truth the Bohr model is a laughably simplistic stick-figure representation of matter. It describes certain behaviors adequately but completely fails to describe others. It was quickly extended by De Broglie, by Schrödinger, and innumerable others to include wave and then quantum characteristics.

Despite its almost laughable simplicity and innumerable refinements and extensions made over the last century, the Bohr model remains one of the most important and consequential scientific models of all time. If the Bohr model was presented in many book discussion groups today, it would be criticized, dismissed, and even mocked as having no value.

Certainly we can and should recognize and discuss the limitations of models. But we must not dismiss them out of a mistaken lack of appreciation of the limitations of scientific models. Often these misguided criticisms have the more widespread effect of unfairly discrediting all science. Following are some examples of the kinds of criticisms that are valid and some that are invalid.

  1. We must first recognize when we are talking about a new idea like memetic evolution, that we are talking about a scientific model.
  2. A scientific model does not need to answer everything. We must recognize the limitations of every model, but the more important focus is on how useful it is within its applicable limits. Newton’s Laws do not describe relativistic motion, but in our everyday world Newtonian physics is still fantastically useful. Critics of science should not claim that a model – or science in general – is fundamentally flawed or unreliable because a particular model is not universal.
  3. Many critics of science think they have scored points by pointing out that “you can’t trust science because their models are always being replaced!” But models are hardly ever replaced, rather they are extended. The Bohr model was greatly extended, but the basic model is still perfectly valid within its range of applicability.
  4. The fact that there are many different models of the same thing is not proof that “science contradicts itself and cannot make up its mind.” We famously have the two major models of light- the wave model and particle model. The wave model correctly predicts some behaviors and the particle model correctly predicts others. Though they appear irreconcilably different, both are absolutely valid. Real light is not exactly like either model but is exactly like both models. Think of your mother. She has a mother-model that describes her behavior as a mother. But she also has a wife-model, a career-model, a daughter-model, a skeletal-model, and many others. None of these in themselves completely describes your mother, and many may seem irreconcilably different, but all of them correctly model a different set of behaviors in different situations and only collectively do they all communicate a more complete picture of your mother.

So, when discussing something like memetic evolution, it is proper and correct to ascertain its boundaries and to critique how well it describes and predicts observed behaviors within those boundaries. But it is wrong and counter-productive to dismiss it either because there exist other models or because it does not – yet – describe everything. And worst is to dismiss all of science as flawed because it puts forth multiple models of reality and extends them over time.

To describe and predict human thinking, Skinner put forth a stimulus-response model, Blackmore puts for forth a meme-model, and I often focus on a pattern-recognition model. These are not in competition. One is not right and the others all necessarily wrong. The fact that there are these three and many other models of human thinking does not reflect any fundamental weakness of science, but rather its strength.

It us unfortunate that far too few people have a sufficiently deep appreciation and level of comfort with scientific models. We must do much better to understand and communicate these subtleties that are so fundamental and critical to science.

 

Music Appreciation

1 Reply

PatI’m flying to Vegas this weekend to catch “Pat Benatar and Neil Geraldo: A Very Intimate Acoustic Evening.” I have been a fan of theirs since the 70’s and am thrilled to get to see them once again. Unlike so many other older artists who simply run on autopilot, Pat and Neil have just gotten better and better with age, reinventing themselves in fresh ways while still remaining unapologetically true to their rock hard roots. No longer having megastar status has actually freed them to be truly great. Rather than being carefully manicured by the contrived glitz of big concert venues, they truly shine in intimate settings where they can stroll out on stage and warmly greet the audience wearing sweat pants and holding Styrofoam mugs of coffee.

My upcoming concert trip inspired me to upgrade my music system. I used to have an admittedly insane $50K worth of audiophile equipment, but like many others I have gradually traded away all that great equipment for the convenience and unlimited access of the digital age. I know that many people are perfectly happy with their crappy ear buds (see here), and are convinced that high-end audio is just pretentious nonsense, like people who insist that a $1000 bottle of wine really is worth every penny. But whether you can hear it or not, to people who can and do appreciate fine audio, you get what you pay for.

While for a long time the very alluring benefits of digital libraries and streaming access have forced us to compromise on quality, the industry has made huge strides. The good news is that today most of us can now get really, really good quality for a very affordable price. You no longer have to invest $20K and dedicate a room to get very good audio that rates at least a B+.

But as a consequence, concessions to digital convenience has pushed the dream of an A+ quality experience even further out of reach. For a while, I feared that truly exquisite high end audio had been essentially killed by digital. But I am pleased that the industry is not remaining satisfied with mass market B+ audio. Streaming services have invested heavily in delivering high quality source material, and the traditional audiophile industry is catching up to provide new digital-friendly equipment that can take those who care enough to the heights of sonic rapture.

Yesterday I visited a local high end audio store here in Tacoma called Advanced Audio (see here). I have to say that the staff there were incredibly warm and friendly, not at all the cliché snooty types that all too often manage these stores. I picked up a Sonos Playbar for my television (see here). The audio is very nice for both music and movies. Certainly not audiophile quality, but excellent for the money with amazing ease of connectivity to all my digital equipment.

Victor, the owner of Advanced Audio, took me into a listening room to show off his combination of Sonos Connect, Auralic Altair, Macintosh MC275 tube amplifier, and small-format Focal Sopra N1 speakers. He let me listen to “Papa’s Roses” by Pat Benatar on the system. I’ve used this track as my reference piece for many years, along with a few others like “Big Love – Live” by Lindsey Buckingham. I know from experience that these songs simply get better and better as your equipment gets better.

I’ve auditioned Papa’s Roses on the best of the best and thought I knew how gorgeous it could be, but I was still blown away this time. I had never heard it reproduced this beautifully. It was like a heavenly orchestra of angels. The song was rendered with such exquisite detail as to bring tears to ones eyes.

I was particularly blown away because the source was streaming digital! It is very gratifying that the high end audio industry not gone the way of vinyl records. Instead, it has met the challenge to bring stunning quality to the digital world of music. Now it is true that if you want to experience this acoustic nirvana, you will have to fork out around $25K. But while out of reach for many, it is still good to know that it is out there waiting for you and yielding technical breakthroughs that will improve everything downstream.

If you decide to audition a high end system, I can offer some practical advise. While an expert like Victor can talk your head off with acoustic science and engineering, here are 10 simple guidelines to help you pick out the best system for you:

  1. Don’t be swayed by specs. They don’t matter in the final analysis. The only thing that matters is the listening test.
  2. Don’t be fooled by complexity. The best speakers I’ve ever found have been simple two-way speakers, not gimmicky contraptions with 38 emitters designed to wow the unsophisticated buyer.
  3. Bring your own source material that you know very well.
  4. Some systems only sound good with particular types of music. Unless you only care about a particular music type, audition a representative range of genres.
  5. Some systems only sound good with sufficiently high quality sources. This greatly limits what you can enjoy. If I cannot still enjoy my beloved but crappy-quality Nina Hagen recordings on a system, I don’t want it.
  6. If the salesperson won’t let you play your own source material, walk away. He or she is trying to cherry pick only that particular music that flatters the system and avoid music that exposes its flaws.
  7. Don’t be wowed by heavy bass. I don’t want a system that sounds like a woofer array in the trunk of a ’79 caddie.
  8. Look for natural balance, dispersion, and clarity that flatters all parts of the music. Every instrument should stand out clear as the clearest bell. You should hear fingers sliding on the frets, breath on the reeds, and the subtle reverberations of an actor speaking in a hallway.
  9. Test it at all volumes. A great system should sound full and satisfying at low volumes. It should also still sound great – and not tire you out – at 100+ decibels.
  10. Don’t be rushed. The only test that really matters is what I call the “can I walk away” test. Many systems will knock your socks off for a few minutes, then leave you fatigued and ready to enjoy some silence. A great system leaves you wanting ever more and more and more. You desperately want to play everything you own. You could listen for hours and not get tired of it. If you cannot stop listening, you’ve found the right system for you.

Claiming that the experience of true audiophile quality music enjoyment is just made up, is like claiming that watching Star Wars on your television is just as good as experiencing it at an extreme digital big screen 3D movie theater. Great music can only be fully appreciated with great music systems. And I am overjoyed that digital and audio technologies are today converging and synergizing to make that experience accessible to more and more people.

 

Our Northern Flicker

Leave a reply

My wife and I recently moved from Manhattan to Tacoma Washington. Although we still love NYC, we were ready for a change from the endless scaffolding and continual roadwork that seems unending and incessant in lower Manhattan. We were frankly tired of being woken up by jackhammers echoing through the skyscraper canyons (in lower Manhattan they literally tear up the streets and pave them over only so they can rip them up again the very next day). We were ready for the peace and serenity of the Puget Sound.

Imagine our Deja Vu shock to be virtually bounced out of bed in the morning by a noise somewhere between a jackhammer and an over-revved race car between 6 and 8 am in the morning. It seemed to come from the general area of the chimney, but it reverberated throughout the house. After exhausting every possibility inside, I went outside to spot a paunchy little bird perched on our chimney, industriously drumming away with his beak on our metal chimney cover at like 10,000 ppm (pecks per minute). He was essentially the little transducer at the base of a huge sound resonator.

northern-flickerIt turns out that he or she is a Northern Flicker and they are well-known to north westerners because they are regionally infamous little drummers. Of course no one can say with certainty why they do this but we can speculate. They are peckers by nature. They peck out hollows for homes with their beaks, they peck to find food, and they peck produce a unique sound that attract mates or communicate with them. Hey, they have a very efficient and powerful little hammer, and when that is all you have…

Some people assume that the bird is just mindlessly pecking on metal because they are too stupid to realize that it is metal. I don’t subscribe to such dismissive and diminutive assumptions regarding animal behavior. This kind of view often arises from a false notion of human exceptionalism that is endemic to religious thinking.

Instead of only taking pride and self-satisfaction in how unique and special we are, I also take great pride and satisfaction in appreciating how alike we are with our animal cousins. Rather than feel diminished by comparisons to animals, by ascribing human-like motivations and capabilities to them, such comparisons give me a deep sense of continuity and familial community with all of nature. Furthermore, we can better learn more about ourselves if we are more open to recognizing our own simplified and less complicated behaviors and motivations in other species.

Therefore, when it comes to our Northern Flicker friend, I think that, like us, he drums for many reasons. Drumming is what he does, he’s really good at it, he takes pride in it, and he enjoys it so it does it just for fun. He probably really, really likes the huge megaphone that our chimney cover offers, and likes to be the loudest Flicker in the neighborhood.

This is not to suggest that our Flicker’s emotions and behaviors and intellect are on a par with ours, but they are simpler versions of our human versions in the same way that his little bird legs gave rise to our human legs and his littler eyes are earlier versions to our human eyes. Their behaviors do not merely “appear” human, they are exactly what evolved into our more complex feelings and emotions. Just as we aren’t the only animals to have some form of brain, we aren’t the only animals to have some level of emotions and intellect and feelings. To dismiss these deep and direct similarities out of some religious sense of separateness is, to me, a highly sad and lonely pedestal on which to place ourselves. You may choose to DEFINE emotions as things only humans have, to DEFINE intellect as intellect only when it reaches human capabilities, but that does not negate the real presence of highly developed precursors in animals.

And just as the drumming of our little Flicker resonates and echoes and touches others in ways he cannot imagine, so too do our more complex behaviors reverberate our to touch others in tangible and deeply personal ways that we cannot imagine. If I were to make it impossible for our little drummer to peck on our chimney cap in some way, he or she might very well start to peck on the wood of our home and that would be much worse for us.

So, my new Flicker friend, you go on drumming on our chimney cap. I grok you and it enriches my life to listen in on your early morning broadcasts. I can identify with your joys and compulsions and frustrations and yearnings for a mate. I hope that later this spring, when your drumming stops, it will mean that our chimney cap has helped you find a mate who will give you other things to do with that spectacular beak of yours!