knadh

I've talked about the technical issues in a couple other responses on this post.

About spam, we've been working with the capital markets regulators, cyber crime units (Police), and TRAI (telecom regulator). It is a nasty issue that plagues all capital markets institutions in India. I'd gone into some detail on the data leaks on this Reddit AMA last year: https://www.reddit.com/r/IndiaInvestments/comments/bhv18e/we...

Thanks :) The online on-boarding is the result of a multi-year effort of convincing the regulators of the advances in tech.

As I mentioned in another response, sorry about that. We've had three incidents (intermittent degradation of service over the last two months that lasted 7 and 30 mins respectively). There were two similar incidents last year.

Not sure what you mean by "HTML errors", but apart from these unfortunate incidents stemming from legacy infra dependencies, there are no issues, and we cleared a billion trades last year.

Not sure if you noticed, the volatility over the last two months caused all top five traditional mega-brokers to go down for hours multiple times over several days. Of course, none of this should matter to end users, but I just wanted to say that a lot of these issues in the industry are because of the ageing underlying infra. Thanks.

Hey, sorry about that. And we're always sorry about technical issues when they happen, and 9/10 times, they stem from systems outside of our control.

The underlying infrastructure in the Indian capital markets is very constrained and was not really built for the kind of scale we're operating at in 2020. The next biggest broker processes only a fraction of our daily volumes. So it really is a matter of scale, industry wide.

To give you an example, the leased lines NSE (National Stock Exchange) offers to brokers to send orders take up to four months to commission, and each line has a max capacity of 1000 messages / second. We've been processing close to 8 million trades a day. So you can imagine how painful things can be.

We've been working hard over the years to reduce legacy market infrastructure dependencies and have been making significant progress.

listmonk supports arbitrary nested tags and attributes in the form of JSON. A subscriber can have properties like {"purchased_x": true, ...}. It's possible to issue complex SQL expressions to filter and send campaigns to subscribers.

The multi-list approach has several other benefits. When manager / sender (and other) permissions get introduced, it will be straight forward to restrict users to managing certain lists. In addition, multiple lists allow subscribers to selectively subscribe / unsubscribe from lists.

Internally, the structure is simple. There is only one subscribers table and subscriber data is not duplicated anywhere. List (foreign key) relationships are in a separate table.

Any SMTP server. SES also offers an SMTP interfaces.

That looks interesting. Will check it out.

Shameless plug: https://listmonk.app (FOSS. Written in Go)

Thanks :) There isn't one in JS, but it should be really easy to port.

Metaphone is a much better alternative to Soundex for phonetic indexing of English words. Phonetic indexing is an interesting concept, and inspired by Metaphone, I was able to write algorithms for Malayalam[1] and Kannada[2], two Dravidian (Indic) languages. They work well for searches and suggestions.

[1] https://github.com/knadh/mlphone

[2] https://github.com/knadh/knphone

We've been doing $0 broking in India for a while now. Nithin (founder) recently shared his perspective[1] on this, albeit largely in the Indian context.

TLDR: While going $0 was long time coming, US brokers have other avenues of earning, unlike their Indian counterparts.

[1] https://zerodha.com/z-connect/rainmatter/the-race-to-zero-ca...

Looks nice. For anyone looking to self-host or develop their own CRMs, I can't recommend https://erpnext.com enough. It's written in Python and is FOSS.

This makes me nostalgic. Spent inordinate amounts of time writing terrible, unmaintainable Perl scripts and FTPing them to cgi-bin on free 5MB hosting plans on F2S and Netfirms. Most hosts wouldn't provide error logs, so there was nothing more than a 500 page from httpd to tell you that something had gone wrong. Debugging anything was an endeavor. Then of course, PHP came along and killed cgi-bin.

If I had to pick one thing to represent the CGI era, it would be Matt's FormMail [1].

[1] https://www.scriptarchive.com/formmail.html

Thank you :) listmonk supports unsubscription and blacklisting. The default template that ships with it has an {{ UnsubscribeURL }} tag in the footer. I just realised that it's not in the documentation (work in progress). Will add.

I can't quite recollect, but I think the thought process was along the lines of "hassle free, peaceful list management".

listmonk does have unsubscription and blacklisting, although not bounce processing (yet. todo.)

listmonk is a replacement for Sendy. You can connect your SES account to it using the SES SMTP interface.

Integration with in-house DBs was a top reason for us.

Thank you. We aren't. We just have dedicated IPs and proper DKIM and SPF records.

Out of the box, it isn't, but it's straight forward to do it with APIs (for individual entries and for bulk CSV imports) or with direct inserts to the DB (the table schema is simple). In production, we've made a small daemon that listens to a PubSub queue published by our CRM to keep the mailing list up to date.

About attributes, every subscriber entry in listmonk can take a JSON map of attributes that can be queried.

Hey, thanks for phpList! We were on it for the longest time, and personally, I first used it some 14 years ago.

We did try several things; Postfix tweaks, SES / Postal SMTPs, rate changes in the phpList config etc., but nothing seemed to work. This, along with the other factors (segmentation, dynamic templating), lead to listmonk.

You can use SES's SMTP interface with listmonk. With some added effort, you can self-host an SMTP server and achieve better throughput, and depending on the volume, cost reduction. We've been running our own Postal SMTP instances successfully.

Thank you. It was a bit of a surprise to me that Sendy had no viable open source alternatives. phpList is okay, but is quite dated and has a number performance issues.

Author here. To give some context on why listmonk was built, at work (regulated financial business), we have to deliver e-mails, mostly important updates, to 1.5mn+ customers regularly. We used phpList for the longest time and then tried MailTrain and Sendy before finally deciding to reinvent the wheel after running into a number of issues, of which, a few important ones are mentioned below.

- Performance. Unreasonably long amounts of time to send out e-mails. phpList degraded to the point of taking several days to process a campaign. listmonk can spawn N goroutines (~threads) and push e-mails to multiple SMTP servers. On a commodity ec2 instance, we're able to send 1.5mn+ e-mails in a couple hours.

- Subscriber imports were extremely slow. Direct integration to keep subscribers in sync with external CRMs was cumbersome. Direct DB inserts were complicated due to the complex table structures. listmonk imports 10k records/sec into a Postgres DB on a commidty ec2 instance.

- Segmentation. Often, we have to rapidly segment users by custom attributes and conditions and relay an update to them. listmonk supports SQL expressions to segment users on their attributes that are defined as arbitrary JSON maps (thanks to Postgres JSONB type).

- Unavailability of dynamic templates. listmonk templates support Go template expressions so it's possible to write logic in messages to make them dynamic.

I would like to add that listmonk is still work in progress and requires a number of essential features before it can come out of alpha.

Author here. We've been using listmonk in production at our company (regulated financial business) to deliver e-mail updates including regulatory ones for over 6 months. We host our own SMTP instances using Postal[1] on EC2 instances and have never had any issues with deliverability. If it's legitimate e-mail, I don't think it's much of an issue.

PS: Postal SMTP is awesome.

[1] https://github.com/atech/postal

I made a similar open source verification gateway recently. It supports emails and you can write other providers as plugins.

https://github.com/knadh/otpgateway

Thanks :) Weekend bars are now blue.

Metaphone [1] addresses a lot of issues Soundex has. While Soundex is aimed specifically at names, Metaphone works for all English words.

PS: Inspired by Metaphone, I wrote MLphone [2] a phonetic lib for the Malayalam (South India) language. The phonetic keys the algorithm produces are Roman characters though.

[1] https://en.wikipedia.org/wiki/Metaphone

[2] https://nadh.in/code/mlphone/

Deciding on a pattern on writing http APIs in Go was a bit of chore. Ended up using the `pat` library for chaining middleware. Quite extensible and light weight. Also, using context to pass objects through the requests chain is a neat trick.

Nice. From my crude and unscientific benchmarks, I've found that Go is able to handle a lot more concurrent WebSocket connections than Node.

Yes, there is no end to end encryption as of now, although there is no persistence or storage of any sorts on the server.

Author here. This is meant to be something super simple and instantly accessible. Start and finish a conversation in mere seconds if need be with no traces.

I am sure downloaded clients with end to end encryptions exist, but it's definitely outside the scope of something as simple as Niltalk.