Personality trait recognition using ECG spectrograms and deep learning | Hacker News Reader

Personality trait recognition using ECG spectrograms and deep learning

arxiv.org

48 pointsPaulHoule40 comments

I'm a junior academic physician with a background in CS whose specific research interest is ECG analysis with deep learning. This paper has some glaring holes.

Apart from the total lack of external validation (which I suspect will never come), we already know that ECGs have features associated with emotion: heart rate, frequency of premature ventricular contractions, frequency of premature atrial contractions. Are these deep learning detectors better than running a simple regression after using something like a wavelet transform to detect these specific features? This matters because if you have a method that is more computationally efficient, more easily interpreted, and more robust to extrapolation to out-of-domain data, why in the world would you go with DL?

Of course the article is completely silent on these questions. If I was a reviewer I would refuse to accept until they added extra experiments to address this.

This is a symptom of a more general problem in interdisciplinary medical research: Papers are written and reviewed either by engineers who have limited understanding of the clinical background or by clinicians who can code a CNN in pytorch but have no knowledge of classic ML methods or ML theory. And so we see seriously flawed papers get accepted. (I know this is a preprint, but I've seen similar stuff in well regarded journals.)

It's incredibly frustrating as someone who tries to do truly rigorous research in this space, only to be desk rejected because ECG magic sounds a lot cooler than incremental gains.

> This matters because if you have a method that is more computationally efficient, more easily interpreted, and more robust to extrapolation to out-of-domain data, why in the world would you go with DL?

One reason we will see more deep learning in electrophysiology is simply because it allows one to abstract over a good deal of statistics and signal processing, which are hard. Deep learning is being used as a black box for those who can't or don't want to understand their data.

I agree with all your points. I think that training algorithms without understanding the underlying data is fundamentally a bad thing. This isn't NLP, where we're willing to put up with LLMs being inscrutable because they blow traditional algos out of the water. Accuracy with deep learning in ECG is moderately better than traditional methods, not paradigm shifting. And unfortunately the drawbacks of DL in ECG analysis are underanalyzed and underdiscussed. Most published DL ECG models that I've played around with have pretty significant failure modes that would preclude them from being used clinically. Of course, it's research, it's not meant to be clinical grade. But the failure modes aren't mentioned at all in the publications, and I've seen very little in the way of research into clamping down on some of these failure modes.

There are far too few neuro + medicine + CS people. Being able to coordinate research in two of those happens now days we need all three.

Part of the problem is CS people thinking the brain is a computer and biology people thinking ml is magic.

I've done ECG research in the past and these ai reads your mind posts always make me very skeptical.

> Are these deep learning detectors better than running a simple regression after using something like a wavelet transform to detect these specific features?

It’s pretty insane to think that a decent neural net trained on sufficient data wouldn’t be able to outperform basic regression.

What makes you think that a neural net must necessarily be significantly better? How much better is significant? Can one actually train a large enough network on (finite) available data? Sometimes it might take god awful amounts of data+compute+layers to rediscover transforms that we have ready access to, once math has matured enough.

It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression. But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression.

> It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression.

Well, maybe it's logically obvious, but it's also empirically false: https://www.youtube.com/watch?v=x7psGHgatGM (Skip to 13:36 to get to the point)

> But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression.

As I've said elsewhere in this thread, a deep neural network of N layers can be decomposed into a complex nonlinear function of N - 1 layers followed by a linear combination of the nonlinear features generated that function. There's no law that says that you have to use a neural network to generate your nonlinear features. You can use any method you like and then linearly combine those.

I don’t think you understand what you’re saying. “A linear combination of non-linear features” is not what people mean when they talk about linear regression. And even if they were, a neural network will do a much better job of generating the non-linear features than you will be hand. So again, it’s logically obvious that a neural network will be capable of performing better than simple linear regression.

> I don’t think you understand what you’re saying. “A linear combination of non-linear features” is not what people mean when they talk about linear regression.

According to who? Did you watch the talk I linked? Because in that video, NeurIPS gave their test of time award to a paper where they generated nonlinear features and used those to train a linear classifier. I guess the committee at NeurIPS also didn't understand what they are saying?

> And even if they were, a neural network will do a much better job of generating the non-linear features than you will be hand. So again, it’s logically obvious that a neural network will be capable of performing better than simple linear regression.

Again, logically obvious but often empirically false.

It's really not. If you throw a blob of unstructured data at a neural network, then sure, it will beat everything else by a mile. If you actually try to understand your data and engineer your features, you can often beat DL with much simpler methods. (Obviously YMMV depending on the specific problem domain.) Unfortunately, feature detection and feature selection are dying arts. Just throw all the data at a CNN or transformer, report the accuracy, and that's it, that's the paper. No analysis of robustness, edge cases, etc. because you can't easily do those things if you haven't taken the time to understand your data in the first place.

There's definitely xor type nonlinearities going on in factor models of personality types. The interaction between two factors is often very different at the extrema of each factor - often flipping sign.

Regression models can't capture that. Now it's possible that the behavioral nonlinearities are somehow emergent, and that despite those, the ECG can be captured by a linear model. But I wouldn't want to bet on that.

Nonlinearities in ML existed long before DL and ReLU. What's at the end of a deep CNN? Probably global pooling followed by a dense layer? That final dense layer is taking a weighted combination of the last stack of feature maps. So deep learning is fundamentally the same as the old approach: Construct features with a nonlinear transformation of the input. Then calculate a score by taking a linear combination of those features. The only difference with deep learning is that the manner of constructing those nonlinear features is left unspecified.

Interesting comment. I'm not an expert in CNNs but what you're saying makes a lot of sense.

Question: are you saying the final layer is "in effect" a linear combination? At least in transformer architectures, the dense end block is iirc three layers deep and uses relu. Even if the CNN's dense final part is one layer deep, wouldn't it also be using relu activations?

Even a one layer dense ANN can capture nonlinearities if it has a nonlinear activation fn. But maybe in practice the activations don't do a lot of work? Or am I simply mistaken about final layer, does it simply have linear activations?

Also, can you share your intuition about the nonlinear features? Spectral/wavelet analysis? Or something more complex?

> Question: are you saying the final layer is "in effect" a linear combination?

Not just in effect a linear combination; it is a linear combination. There are some exotic nonlinear NN layers that are used in particular niches. But in general, NN layers are syntax sugar over matrix multiplications (i.e. linear functions). The nonlinearities are the activation functions between the layers.

> But maybe in practice the activations don't do a lot of work?

No, the activations do do a lot of work. There are real nonlinearities in the network.

> Or am I simply mistaken about final layer, does it simply have linear activations?

Sometimes you actually do use a linear activation on the final layer for certain regression tasks, but that's not the main thing I'm getting at.

Let's say that your final layer has is a ReLU activation. What is this conceptually? You are taking a linear combination of the features from the previous layer and them clamping the result to >= 0. Sure, that's a nonlinearity, but it isn't going to have much in the way of emergent modeling capabilities. You need to stack many, many nonlinearities before you get that.

So my point is that a deep neural net of N layers boils down to a complex nonlinear function of N - 1 layers, followed by a "dumb" linear combination in the final layer. You can do this with traditional ML methods as well, but you have to handcraft your nonlinearities.

> Also, can you share your intuition about the nonlinear features? Spectral/wavelet analysis? Or something more complex?

There's innumerable possibilities here. It could be starting with a method that's inherently nonlinear, like nonlinear PCA, polynomial regression. Or it can involve transforming the output of a linear function (like the Fourier transform) in a nonlinear way.

Admittedly this very tough. And for really tough problems, like video synthesis, effectively impossible. But NNs get thrown at much simpler problems all the time.

If regression on wavelet coefficients solves your task then sure DL is unnecessary.

I spent half a decade on a data science team developing algorithms for ambulatory ECG monitoring at scale. 100+ human years of 256 Hz multichannel ECG collected everyday. Waves, beats, & rhythms all segmented and classified. PDF reports generated for cardiologists and EPs.

DL techniques like unsupervised pre-training blew away every other classical or purely supervised method we ever tried. Especially for rare and critical arrhythmias like VF. It unlocked value in massive unlabeled datasets. It gave us a framework for scaling model performance as a function of dataset size and compute, with no fundamental limit.

I do get your frustration about the publication game. You might find industry is more aligned with pragmatism & efficiency.

>The ASCERTAIN dataset comprises a diverse range of physiological signals, including ECG recordings, collected from 58 participants exposed to video stimuli (36 videos) categorized in different categories based on valence and arousal levels. In particular, there are four subcategories of these 36 video clips. Clip 1 to 9 is categorized into High Arousal and High Valance (HAHV), Clip 10 to 18 Low Arousal and High Valance (LAHV), Clips 19 to 27 Low Arousal and Low Valance (LALV), and 28 to 36 High Arousal and Low Valance (HALV) clips [10]. ECG signals from the right and left arm were recorded at a sampling rate of 256Hz… our model outperformed the closest rival by a wide margin (0.56), achieving an accuracy of 0.94 for the extraversion trait. Similar trends are seen for agreeableness (0.92 versus 0.55), conscientiousness (0.92 versus 0.60), emotional stability (0.93 versus 0.53), and openness (0.93 versus 0.48).

Predicts Big 5 based on heart beat response to various stimuli. I wonder if EKG’s really contain more information beyond heart rate as the conduction within the heart is pretty consistent normally, regardless of rate. Neural nets seem to work great at finding any signal- too bad it isn’t so clear what those signals are.

I can't tell if they are using a 2-lead or 3-lead ECG. Personally I think it's pretty cool that a 12-lead EKG can see the electric field generated by your heart as a vector

https://en.wikipedia.org/wiki/Electrocardiography

I've had health care professionals wire me up for a 12-lead ECG (always normal so far) in less than a minute too.

Can anyone predict a potential benefit to society that might come from this research? I can only think of bad things coming from this. It's like a major step down the staircase towards dystopia.

Everyone was up in arms the other day about a developer who had to take a personality test for a job at FedEx. Imagine what this leads to.

The researchers are arguing that using ECG reduces subjectivity in personality assessment. Which is predominantly determined by self reported questionnaires.

They are mapping ECG data to the 'big 5' personality traits, and neglected to mention how these traits were determined for each participant prior to measuring the model's predictive accuracy. Almost certainly these primary trait scores were determined through a questionnaire.

Does this have any more validity than the E-meter from Scientology?

From the paper:

>Traditionally, assessment for personality traits has relied heavily on self-reported questionnaires and behavioral observations. While these methods have proven valuable, they also introduce subjectivity and potential biases

Phrenology with a few modern buzzwords thrown in.

I can't wait for my next job interview where they give me an EKG.

Hah, the future is the other way around - company runs EKGs on teams and extract a vector per team, pick vectors for teams with best stack rating or some other gamed-to-death "productivity metric".

Hand vector to recruiting agency and have them run EKGs on the desperate masses. Hire copies of your best team. Profit.

Season with other "data points" like DNA tests.

"Whenever the person makes an error, a special pattern of brain activity shows up: a sharp, negative electrical activity that is strongest at the top of the head. Since this electrical activity is negatively charged and associated with making errors, it is called the error-related negativity or ERN'..."A curious thing about the ERN is how quickly it happens after you make an error. So quickly, in fact, that it happens before you are aware of your mistake. The ERN usually occurs no later than 100 ms (1/1,000 of a second) after an error has been made. " https://kids.frontiersin.org/articles/10.3389/frym.2020.0008...

So if I wear one of these, I can get a desklamp that flashes when I make a mistake, it's going to be like a strobe :)

Using the spectrum of the ECG is a non-starter unless the ECGs fed to the DL algorithm (after the model is trained) for detecting the personality traits are obtained under the same conditions of the original datasets (while watching a short video, sitting on a chair, etc.)

The spectral content of the ECGs will change depending on the heart rate. So if you run the DL model using an ECG collected while the subject was, say, running will generate completely bogus results.

Very nice. Now let's see the external validation.

Would be nice to see. Doesn't look like a hard experiment. Even a 12-lead ECG is pretty easy to set up compared to an EEG, PET scanner, or something like that. There's a very direct line from the heart to the sympathetic and parasympathetic nervous systems so it's quite believable that at least some of the "Big 5" should manifest in HRV. The reliability of their test doesn't seem far off from conventional personality tests

https://www.psychologytoday.com/us/blog/people-unexplained/2...

Did I miss something or did they not hold back any validation data? The whole thing would be hopelessly overfit, correct?

They use 10-fold cross-validation, but yes imo this entire paper does not pass the smell test. They used pre-trained Resnet and ViT without specifying pre-training on what so Ièm 99% sure that they picked their weights from the pytorch model zoo or similar and as applying an ImageNet-pretraining to ECG classification.

Know it's a loose analogy.

But we really don't know how AI works, really, when we peer into the memory.

And earlier today someone posted how to Visualize internals of Mistral 7B.

And we don't really know how brain neural net works.

And here is a study peering into it.

And in both posts, the visual output is very similar.

They are both neural nets really.

https://news.ycombinator.com/item?id=39378773 https://github.com/valine/NeuralFlow

I really hate when people post idealogical nonsense like this when looking at cross-displinary work. It almost feels like a religious level of arrogance to make these comparisons.

There are some interpretability problems with modern machine learning models but we have perfect information about how they work. You can observe their state directly and the math backing them is very well understood. Contrast that with the brain where observability is extremely limited and how it works on the tissue scale still isn't that well understood.

The similarities are surface level and, imo, a distraction to making serious progress in both fields. Talking about how these things mimic the brain is more marketing than science.

1. Where was there any reference to any 'ideology'?

2. Cross Silo ("disciplinary") work is often misunderstood, rebelled against.

3. "Perfect Information" isn't understanding. "Observing State" isn't understanding. See the dozens of articles on AlphaGo.

4. Yes, there is still a lot to learn about the brain. That doesn't mean we don't already know a lot. There are neurons right? There are electrical potentials right? It exists in the physical universe right? It can be modeled right? So when do the models scale up enough to match reality? You can't say 'Never'.

The ideology of your views exists whether you're aware of it or not.

Your points 3 and 4 are contradictory. You simultaneously assert that AI models that we have a direct view of are these mysterious and unknown things while making the opposite point about the brain, an object that stresses the limits of computation.

Point 3 and 4 are not contradictory. I'm just not being clear. And I am not being 'mysterious'.

For the brain, let's suppose we can have as 'perfect information' about 'state', as we can have about a computer neural net.

Then

Either they would be equally 'mysterious', we would still not know how they 'subjectively' feel when making decisions. Where is the 'experience' coming from. Like with humans, where do 'thoughts' arise from before you think about what to think about (going back to Schopenhauer). Like with AlphaGo, can you really find the point when it 'decided' on Move 37, or is it a process with pre-curser neural "firing's" same as a human has.

In both, we have perfect information of state, and further, do determine some correlates between a 'state' and a 'feeling' or 'process' and actions. Let's suppose as you imply we 'understand' everything about what is happening in AI. What I am say, this will eventually happen for humans, that we'll understand the brain. There wont be any mystery left.

Since for both we would have perfect information about state, and we can figure out how the processing is happening, and we 'understand' both -> THEN we would have to accept that AI is also as conscious as a human.

---

I'm just saying the Silicon-Carbon argument.

Either we prove or understand that Humans and AI are both Conscious. or Neither are Conscious.

Once we completely explain away human thought, we'll find we are 'clockwork' machines also.

But, yes, in future, and any rudimentary comparisons right now are probably a leap. Just I don't think it is as far a leap as you are implying.

Neural nets were literally modeled on brains, why can't we make comparisons.

> Neural nets were literally modeled on brains, why can't we make comparisons.

They're quite a shitty model that comes from a 1950's understanding of neuroscience.

Think in AI and Neuroscience, there has been some progress since 1950.

This is the new shit.

Measure some kind of data which most people don't fully understand (e.g. ECG spectrogram), label it with arbritrary bullshit (skin color, political beliefs, COVID positive/negative), apply some kind of ML/LSTM black magic, add some data, subtract some training data, so you get a more or less good fit.

Publish the paper, bam! data science/machine learning!

I can't wait for this technology to be used by HR. /s

Reply on news.ycombinator.com