rachbelaid

With 2ndquadrant working on Postgres-XL (http://www.postgres-xl.org/), I think that you can be confident that you will see a lot of the features being proposed to core postgres. It will just take some times to build the building block necessary like: global index, distributed sequence, repartition ...

I quite confident that the postgresql from 5y in future will be quite different in term of storage / server topology support. I won't be surprise pg_bouncer capacity to finally make its way to core when we have a coordinator.

Postgres has steady progression (even if not fast enough for some people) but they are moving without compromising robustness of their product for the users.

Some work started in this direction. I didn't follow closely the whole thread but I don't think that got commited in PG10.

https://www.postgresql.org/message-id/flat/20170119213859.GA...

More info in the EDB roadmap: https://wiki.postgresql.org/wiki/EnterpriseDB_database_serve...

Postgres has been amazing in shipping the foundation required to deliver complex feature.. Logical Replication is an example of it, all the piece commited in the last 6y allowed to make this patch achievable.

The conversation on the patches are really interesting: https://www.postgresql.org/message-id/flat/55D3093C.5010800@... https://www.postgresql.org/message-id/flat/ad16e2f5-fc7c-cc2...

A quick update to say that few things got added since version 0.1 and you can find more details on the github release page. https://github.com/rach/pome/releases

(Author here) Thanks for the comment. I used Grafana and it's a great tool. One of the motivation behind this project is to be Battery included. A tool like Grafana require a timeseries DB (graphite or influx) then you need something to collect the metric like collectd and maybe an aggregator like statsd.

I want Pome to be simple to run as binary.

I did discuss the idea of allowing to disable the web and support pushing data to existing tools. I've been quite busy lately but the project is not dead.

Haskell does but I'm comfortable in Go and easier to get people to contribute

Here the author. Glad and scare that this project reached the 1st page of HN. This project is at a very early stage but I had to release it at some stage. I wrote some explanation in the Readme [1] and in a blog post [2].

TL;DR: Pome aim to be very simple to deploy, opinionated and battery included tool to have a look at the health of your PG db. It's maybe not the case of anybody here but in my career, I have seen many PG db for which no health status were track (or because people think that RDS is magic). I assumed that if a very simple tool existed then it gives less reason to not track their health status.

At this stage, I don't think that Pome offers enough to be very useful but I hope that you will like the direction taken and where it's going.

Pome isn't aiming to be a tool for humongous Postgres instances which are already in the hands of a DBA who can have the time to setup more advanced monitoring tools. Pome won't be an alternative to a more configurable tool like collectd.

[1] https://github.com/rach/pome#why-building-pome

[2] http://rachbelaid.com/introducing-pome/

Yes, it supports RDS and I want to support it as it's the one that I used most often. It's really basic right now but I will add some new metrics this month.

Here the author. If you dislike Go then that makes two of us :)

But more seriously, I wrote a bit about the why in https://github.com/rach/pome#why-building-pome

To keep with the simplicity that I was aiming, I wanted a binary. I considered Rust, Haskell or Go (Swift was not opensource yet at the time) but went with Go because the libs that I will need (cron like scheduler, embedding assets, etc).

I had never written something in Go before but if I've known go then maybe I would have pick an other language as I started this project with a learning motivation.

Thanks you. typo mistake.

Thanks for sharing this. I will look at which metrics that I can include from this project. I can also say that CPU can't happen to keep the project gathering data only through Postgres (except if you know a way)

The goal of Pome was to be very easy to setup for people who put nothing in place, which is why wrote it in Go. But if you have some time there is much better/complete tools like collectd.

Here the author. I understand your points, I wrote a bit about my personal motivation behind this project. http://rachbelaid.com/introducing-pome/ I also wrote within the readme about why wrote the project (context+goals), https://github.com/rach/pome#why-building-pome This project is not aimed to replace more advanced tool like collectd but to offer a basic all-in-one solution. I'm also not planning to support huge database for the scaling problem that you highlight. Trying to offer something easy without RDD store constraint for people who right now doesn't monitor their db. I hope this make the goals behind the project more clear.

SEEKING WORK - San Francisco - Senior Software Engineer

Experienced Python developer with a good database knowledge, with experience in Ops and Javascript.

- Programming: Python, Javascript, Go

- Frameworks: Pyramid, SQLAlchemy, Django, React

- Software: Nginx, Postgres, Redis

- Experience: 10y

- Experimenting with: Swift, Haskell, Elixir, Phoenix

Download my resume and read more about me here: http://rachbelaid.com/about/

Blog : http://rachbelaid.com/

Github: https://github.com/rach

I found it a good reading on how the community has been adding all the foundation into postgreSQL for complex features

There is also a patch to bring columnar storage into 9.6 (coming from the Axle project, EU founded). Definitively the gap becoming smaller but CitusDB is also having a lot of the glue built already with easy sharding (pg_shard) .. It's an exciting time for PostgreSQL!

From what I've been reading on the on the development of 9.6 (https://commitfest.postgresql.org/), we may only see parallel seq scan and ordering coming to 9.6. They added the foundation to make more thing parallele but I don't think that we can hope to see parallel aggregation coming into 9.6

I had a similar experience. And event after living in a UK/US for 6y and my partner being a native english speaker, sometimes I struggle. One other frustration that I had, when you have to argue for your case. If you have to justify a solution against a native english speaker that can create some frustration as finding the right wording for the arguments can be sometimes a challenge.

Coming from Belgium, petke is also right about the influence of small language community. In example, in the flamish part (where belgian speak dutch), a lot of tv program and cinema is dub from english and even some cartoon. Where in the Walloons part (where belgian speak only french), we have access to the big tv network from France so we never watch anything in English.

But I remember reading that different languages use different sounds spectrum and hearing/understanding languages outside your spectrum is harder. Some languages have overlapping sounds spectrum which make learning different language easier. Now you have also the languages sharing the same roots and as you said somebody who know a germanic language then will have easier to guess/learn new vocabulary.

It has been tested and discuted during a long time. Check the thread if you want to know more about this. http://www.postgresql.org/message-id/flat/CAA4eK1KTv73uD9_W5...

Keep in mind that's the first pieces and not all the patches has merged yet concerning this feature. Still some on progress.

"Batteries Not Included" in this article refers to the fact that PostgreSQL doesn't offer any built-in solution yet to do HA (automatic fail-over) and you have to depend of 3rd party solution.

I agree with you. Pyramid is one of the most well designed codebase that I know.

(IMHO) Complex code is difficult to test so if the code can be tested easily without over using factory or tight coupled setup required then maybe the code is not so complex and maybe it's just an hard problem. Writing test can help to highlight an over complex code and an alternative code will have much simpler tests. It's not a metric but it's probably easier to agree on clean test

Great post with some very useful insights, thanks also for the links to the issues. I do like the approach of the coreos team of working collaboratively on a spec to include every platform. https://github.com/appc/spec

I'm flattered :)

I think that you nailed it, Elasticsearch JDBC is not flexible and you need to be good in SQL if you have a complex model with many relation. Because JOIN create a cartesian product you can end with duplicate and you have to find a way to avoid that because duplicate affect the ranking. In my case I ended using UNION queries.

Also you cannot also have multiple queries but not to update and existing document(eg: adding extra fields) The SQL query for me ended being ~100 lines but the model is quite complex with multiple languages support.

Following the project for which I wrote this blog post, we recently moved to ES mainly because of Japanese support. It's an important point to highlight and not all the internet is english. I was satisfy with PG for english, french, spanish but PG extension for Japanes was not looking inspiring

I don't know why and I can't help you, in both case I didn't post it. But we don't have multiple url for our blog. I find it odd to be honest as the post was posted on HN the 29th of September (base on Google Analytic)

It's also a Django project, I like what you have done. It's a nice lib and it's intuitive. How do you handle a ES document which is a composition of multiple Django Model?

It's just personal opinion but I try to avoid having my application responsible of the data integrity so I went into the way of ES River plugin to pull data. I used the JDBC one, https://github.com/jprante/elasticsearch-river-jdbc ... I met some problems but at the end, it works quite well and don't need logic in the application keep the data synced.

Two other reason that I used River was to not make the app slower by saving data in ES and also of being able to run the application during development without the need of ES and being installed able to substitute the search with a stub.

To be honest, I'm quite surprise to see that somebody posted this article again.. I don't know HN rules when you can post again something.

I'm not going to complain about it :)

As the author, it may be good that I clarify the goals behind this post.

First I'm glad that this post made HN first page for a second time.

I encourage to read this comment because it summarizes well few of the missing features in Postgres about search. https://news.ycombinator.com/item?id=8715624

It's not the first time that I get similar feebacks and I understand that "Enough" is subjective notion and maybe I should've pick an other title like "Postgres full-text search for basic search need".

Just to clarify, this post is not against ES or SOLR. This tools are great and really powerfull but obviously this tools will better than Postgres Search. ES / SOLR are tools build to be anly around document store and search and on the otherside the search in Postgres is only one feature in many.

I think there is multiple reason when you would want to give Postgres search a go:

- You have no experience with ElasticSearch or SOLR

- The project is still early stage and you don't want to invest time/money in a secondary datastore (eg: ES cluster)

- You want to give quickly a search feature (maybe not perfect) in the hand of your user

- Team knowledge, you may want to be careful before introducing a new tech to your team and ensure that people understand it. There is nothing worst than introducing a tech and being the only one who can maintain it.

The tech world is big and doesn't only apply to Startup usecase, I worked in very different environment: Financial institution, Open Source company, Startup and Service company. A thing that I learn is that you don't always have flexibility. In some company having new servers is long process and even if I hate you have to do with this constraint then you have 2 choices front of you: You do nothing and wait or you move yourself to offer to your users a workaround solution without any extra dependency ... That's what Postgres search can be in some usecases.

From my experience, if you are already using Postgres and you find the right documentation (this post try to describe a start to end example) then I think you could get a search working in 1 day of work without any extra servers or resources.

I don't have any experience with SOLR so I will have to pick ES, but if you never used it then you are going to have loads of "fun":

- Sync. ElasticSearch IMHO cannot be used as a cannonical datasource so you will need to figure out a sync strategy. Some people may argue on that but I don't think that ES started with this goal in mind. They may change direction now but still I wouldn't store my data only in ES. http://aphyr.com/posts/317-call-me-maybe-elasticsearch

- Hosting. In the case that you are not using a hosted ES service then you will need to learn to host and build a cluster and also monitor it.

- Learning to use it. After reading ~50pages of doc then you start to figure out how to use type mapping, custom analyser and the 42 ways to do a search query.

I can only speak from my experience but it took me ~2weeks to get my head around all this point without any previous experience. The project for which I used PG search and wrote this blog post recently moved to ES. More or less, 1y later and I'm glad that the PG solution was good enought until now. In our case we needed to support more languages: Japanese, .. and I also wanted to get rid of a bunch of triggers.

The results are good and very fast but for english the results are quite similar to Postgres solution. I think that's mainly because I'm using much only ES basic feature, but we have loads of room for improvement which is a good position to be.

Now there is few other usecases where you may want to consider PG search:

  - you are using MySQL and you are using the builtin then moving to Postgres can give your search a bit more power

  - Side project using Postgres and you want to start a search feature.

  - Search on machine output, case where analyser doesn't really matter and you don't need stemming. Lucene features for tokenization and stemming are great.

To summarize, I don't think that PG search is a silver bullet but it may be "Good Enough" for your need to get you started. Spend 20min and read the post and then maybe it's also good enough for you. On the other hand if you know already ES/SOLR and you got the resources then it become less relevant to use Postgres Search. ES/SOLR (tools base on Lucene) are amazing and really feature rich, if you have important search need and long term plan about search then you should look into them.

If you are interrested about the PG search and you want to know more about it than I suggest you to have a look to this blog posts. I think that the author did an amazing job.

http://shisaa.jp/postset/postgresql-full-text-search-part-1....

http://shisaa.jp/postset/postgresql-full-text-search-part-2....

http://shisaa.jp/postset/postgresql-full-text-search-part-3....

I don't know if people will be interrested but I thought few times about writing a following blog post "When Postgres search is not Enough" to describe how to integrate Postgres with ElasticSearch with JDBC River to pull content from the DB.. Let me know if it's something that you will be interrested about.

Sorry for the numerous grammatical mistakes but I'm not a native english speaker and I don't want to spend too much time reviewing my comment grammar

Thanks again to upvote this post on the 1st page of HN