yasserf

Yeah thats pretty much it! Less code in the functions -> less inconsistencies with the db. And for example removing an enum in the db would fail all your front end components using it, so it's really across the whole pipeline.

But, the jist is, we APIs like this (this can be improved with less generic json schemas):

  export const updateSamf: APIFunction<UpdateSamf, void> = async ({ database }, { samfId, ...data }) => {
    if (Object.keys(data).length === 0) {
        throw new InvalidParametersError('No fields to update')
    }
    if (data.title && data.title.length < 3) {
        throw new InvalidParametersError('Title needs to be at least 3 characters')
    }
    if (data.tags) {
        data.tags = data.tags.map((t) => t.toLowerCase())
    }
    await database.crudUpdate<Samf>('samf', data, { samfId }, new SamfNotFoundError())
    if (data.cover && previousCover) {
         await content.delete(previousCover)
    }
  }

  export const routes: APIRoutes = [{
    type: 'patch',
    route: StudioRoutePath.SAMF_CRUD,
    func:  updateSamf,
    schema: 'UpdateSamf',
    permissions: isSamfOwner
  }]

But short answer is we just use node-postgres. If the get/update/delete/insert is a simple fields/table/where we use a helper function. If it's more complex we write custom SQL and use the following approach:

database.query<DB.User & DB.Pet>(` SELECT * FROM "pet" LEFT JOIN user on pet.owner = user.id `)

For one of my projects I check in the generated types and replace 'unknown' (from jsonb) with the specific object manually. That way the concept of any or unknown gets pushed mostly out of the codebase (for simple crud. When joining we use normal SQL queries with some helper tools that verify the fields we are picking exist). Hopefully will manage to figure out how to do with with postgres comments at somepoint.

For example:

  entity,field,type,constraints
  delivery,arrived_at,Date,{ before: 0 }

  form,field_1,field_2,...
  delivery_details,left_at,arrived_at,...

It's not a magical pill, renaming, deleting and custom migrations need to be dealt with manually. But for the most part 80%_of our changes are additions and this way it just works.

I started using schemats because it's a single CLI interface and gives me all I need. I put together a few tools for a typescript heavy backend development experience and one of my requirements was that each one would be powerful enough to achieve my goals + not powerful enough to trip on how to use it (or in schemats case, patch it).

Currently schemats and ts-json-schema-generator got me there, only issue was schemats was a few years out of date so the types weren't as friendly.

So the code itself with typescript is here:

https://github.com/vramework/vramework/blob/main/backend-com...

But the general gist is we have generic crud operators that are away of what the table are and based on that know the types of what we are returned.

  const {viewId } = await database.crudInsert<UserJournal>(
  'user_journal', 
  { viewId, userId, srcOgg: src[0], sprite: JSON.stringify(sprite), duration }, 
  ['viewId']
  )

  await database.crudUpdate<UserJournal>('user_journal', { srcOgg: src[0], sprite: JSON.stringify(sprite), duration }, { viewId })

  const { srcOgg, sprite, duration } = await database.crudGet<UserJournal>(
  'user_journal', 
  ['srcOgg', 'sprite', 'duration'], 
  { viewId, userId }, 
  new UserJournalNotFoundError()
  )

As long as their database in docker is being incrementally updated (using the same migration scripts as on production which only apply new ones) is this hit only when spinning up a clean docker image?

I'm considering exporting an SQL file whenever we run the migrations for our dev tools / tests to use the one large file. But I haven't found it as easy as export -> import to try it out yet.

I usually run a postgres db (as a docker image on circleCI) when I'm running the build. I'm not sure how scalable it is (currently we have a hundred migrations and it's a few seconds so not bad).

The flow is:

- run migrations on an empty DB - create the interfaces and enums - run typescript - derive all the json schemas from the interfaces for API validation

This means we gitignore the actual DB Types which is great in order to reduce the amount of sources of truth and deal with possible issues

This all runs as a pre-commit hook as well. I guess by doing so we sort of validate the SQL is legit via postgres itself. The only issue we face after a commit gets passed the setup stage and tries to run the migration itself is if theres a data* inconsistency we weren't aware of (trying an enum value we thought wasn't used but was). But that sort of things requires special tasks anyways and the migration fails so nothing ever goes wierd (which is why I <3 SQL migrations )

I'm also massively in favour of having the db as the source of truth. When I first started looking into building my own SaaS I actually tried to go down the database first design, where most of your functionality runs within postgres triggers. But it wasn't near as easy to do things and writing SQL logic is much harder to test.

Figured the next best thing is to just tie the database in extremely closely using typescript into the backend (I rarely define custom types, always try to use Pick/Omit/Partial combinations all the way to the client SDKs).

I ended building generating forms based on the meta data in postgres (select fields know the values from enums and multiselect from enum arrays), so if you define a table in postgres, you just specify which fields you want to render (out of the available columns) and in what order and it pretty much gives you a UI form that is directly rendered in react. Works really well when you have a form component library at your disposal.

That being said given how lightweight the actual package is I would be more than happy to address those issues when raised.

Just wanted to pitch in with some of deepstream features.

Deepstream currently has its own JSON/protobuf protocol that is similar to WAMP but also has presence and data-sync built in. The downside is that to get the full feature set you need Deepstream SDKs (which didn’t work out since they take too long to create and our core C SDK attempt for multiple languages failed miserably). So currently it’s just a very mature JS/node library.

Permissions and authentication are also built in. Every single message that goes through the system (subscriptions and payloads) can be permissioned either through custom code or using a custom library called valve, which allows you to provide a config based permission file. You get access to the topic, payload, action, user session, and can even do lookups on the cache for data sync cross referencing.

I spent a while after the company behind it (deepstreamHub) shutdown and released V5 under the MIT license which was an almost complete rewrite in typescript that allows you to:

- add an adaptor layer for any real-time protocol that allows multiple protocols to talk to each other. So in our case was MQTT / JSON / Webpack / HTTP all supported on the same server chatting to each other.

- a complete redesign of meta event listening. Which means any client can get notified when a user subscribes and unsubscribes to a topic (whether an event or a record (data sync object), as long as they have permissions to

- a completely customisable plug-in architecture that allows you to hook into internal events.

- a monitoring plugin that allows all message counts, errors, invalid permissions and general stats to be accessed via a http API

- a fully working cluster that allows each node to be aware of the subscriptions on other nodes for smart data distribution (rather than a very noisy message bus)

- real-time filtering on both rethinkdb and mongodb as a separate process. This allows you to automatically filter tables based on push notifications

And a bunch of other features (multiple database, caches, message bus and other plugins).

Unfortunately the tech itself never really picked off, I think the custom protocol was a large downside since it required custom SDKs and the VC backed nature doesn’t go hand in hand with OS, as mentioned in other comments.

I stepped down as a maintainer over the last year as I found it hard to justify the cost of maintaining such a large feature set with such low adoption. It’s hard to say goodbye to a project that you spent so much time on, and I sometimes wonder if a rebrand and new documentation is all it needs. Adding wamp support can be done in a couple of days, but I don’t think lack of features is the issue behind its lack of success.

Shout out to Wolfram Hempel, the original author of deepstream, and Caplin (a UK company that provides a real-time server called Liberator that powers a few large banks trading UIs) where the inspiration for deepstream was heavily based upon.

Anyways, thanks for reading this far! If you came by deepstream before and didn’t use it for any reason, would love to hear why so I can improve any future project attempts.

marta. provides software to connect caregivers with families across Europe. We believe that it is a right to receive proper care and age in dignity. In our own families, we have seen that the current products and services are deeply intransparent and fail to deliver adequate quality.

We decided to eliminate all intermediaries, connecting families directly with independent caregivers whilst providing both sides with software solutions to facilitate a simple and successful caregiving journey.

Tech used:

- AWS / Serverless - React / NextJS - NodeJS - Postgres

You would be part of a small core team. The position is fully remote, but if you want an office life there's that option in Berlin.

We are looking for a developer with a love for react frontend apps, clean design and a team player.

Feel free to reach me at yasser@hallomarta.de for more details!

marta. provides software to connect caregivers with families across Europe. We believe that it is a right to receive proper care and age in dignity. In our own families, we have seen that the current products and services are deeply intransparent and fail to deliver adequate quality.

We decided to eliminate all intermediaries, connecting families directly with independent caregivers whilst providing both sides with software solutions to facilitate a simple and successful caregiving journey. We do so with genuine care for the families that book our services and love for the caregivers that deliver them so diligently. By taking the hassle and uncertainty out of finding adequate care, we enable families to once again enjoy the time with their loved ones.

Tech stack: React, Postgres, AWS (100% match not necessary)

Apply here https://hallomarta.join.com/jobs/2090870-technical-lead-seni... or get in touch directly: yasser@hallomarta.de

The other aspect is live journaling, which means when a user is asked something they can reply verbally and it automatically records the response.

I’m currently at a stage where the whole web app is pretty much complete, will be adding in subscriptions, mobile apps, programs and a teachers portal in the upcoming months.

Currently looking for someone to help with marketing and sales. Given it’s entirely bootstrapped hiring someone is not yet an option. Would love any form of advise and discussions on next steps and possible collaboration.

The app is live at https://samarambi.com, but still in beta so expect some odd behaviour now and then.

PS. Just launched (soft launch) on New Years so content is pretty much minimal!

You can contact me on yasser.fadl(@)vlandor.com

Looking for a freelance position. Have a decade of experience building complex fintech real-time web apps. Lots of experience with react/redux, react native, nodeJS, angular, selenium flavours, Postgres, redis, AWS, terraform, Redis and more.

Co author of os project deepstream.io Founder of samarambi.com

You can reach me at yasser.fadl@vlandor.com

Remote: Yes

Willing to relocate: Yes

LinkedIn: https://www.linkedin.com/in/yasserfadl

Email: yasser.fadl@vlandor.com

Github: https://github.com/yasserf (main work done on https://github.com/deepstreamIO)

Tech: Frontend, NodeJS, AWS, Devops, Realtime, Terraform, Azure

Don't have a CV up to date I'm afraid (hence LI only). Co-founded hoxton one and deepstream.io. Worked for a decade in the realtime space (fintech / startups) and open to a part-time job or contract, specially around large webapp and devops best practices and deployments.

Looking for a freelance position. Have up to a decade of experience building complex fintech real-time web apps. Lots of experience with react/redux, react native, nodeJS, angular, selenium flavours, Postgres, redis, AWS, terraform, Redis and more.

Co author of os project deepstream.io Founder of samarambi.com

Your can reach me at yasser.fadl@vlandor.com

My thought pattern was more around whether I was/am meant to reach out to the party whose letter I got to inform them so that they can make a decision on how to respond, as it doesn’t impact me personally at all.

Basically I'm not certain how to pitch this exactly. I feel a bit of condolence from the fact that when it was a startup with over ten employees we had the same issue (specially since I was the tech guy)

Anyways, more than happy to hear any suggestions! I been involved with this project for over 3 years so I definitely have a biased view on trying to see the project for the first time.

To compare to other frameworks out there:

featherjs: A framework for real-time applications and REST APIs

meteor: THE FASTEST WAY TO BUILD JAVASCRIPT APPS

deepstream: a fast and secure data-sync realtime server for mobile, web & iot

The problem is that once you provide more functionality then pub/sub you end up in this vague toolkit statement land.

I would assume that the issue is more around the fact that the term data synchronization is still not used as often and so doesn't give the same familiar feeling people get when looking at other pub/sub frameworks. Deepstream supports HTTP/MQTT/Binary and JSON websocket protocols for rpcs, records, events and presence. Maybe that should be the sentence!

A secure scalable realtime server that supports HTTP, MQTT and WebSockets for rpcs, data-sync, pubsub and presence functionality

The issue with that however is taking that approach means we lose the NodeJS ecosystem support which is massive (and required to add custom plugins/maintain the server). We tried it before and it was terrible in terms of metrics and participation.

Thank you for the feedback though! I will definitely be looking at redesigning the home page and will use your feedback when doing so!

Ideally you would just configure it using a config file for your permissions (https://deepstream.io/tutorials/core/permission/valve-introd...) and use http auth to authenticate your users (https://deepstream.io/tutorials/core/auth/http-webhook/).

You can see this sort of configuration here https://deepstream.io/guides/live-progress-bar/

If you want you could add a custom plugin that would take the HTTP server within deepstream and enhance it using express for all your non-deepstream routes, if you raise an issue for it I think it's something that can be done in the near future! But generally wouldn't recommend for non-pet projects since they serve two different purposes.

Edit: Also thank you!

Generally deepstream is an alternative to using firebase, socket.io, featherJS and meteor. However rather than putting the logic within the server you instead run them as microservices and allow deepstream to handle load-balancing for you.

The two main things deepstream offers are:

- Events

You can publish and subscribe to events by simple doing:

deepstream.event.emit('event-name', data)

deepstream.event.subscribe('event-name', data => {})

Which is the functionality you expect to see from most pubsub mechanisms

- Records

Records is events with persistence. They are more heavy weight objects that persist their content in a cache + db and across all connected devices without the user/developer having to actually do anything. The also support offline support if you want to be able to get/set data while offline.

// get a record

const record = client.record.getRecord('unique-data-name')

// wait for record to be loaded from server

await record.whenReady()

// subscribe to any changes that happen in the future

record.subscribe(data => console.log('someone updated this object!', data)

// discard when finished

record.discard()

You can also set data directly without having to subscribe using

client.record.setData('some-record', newData)

The are multiple other aspects deepstream provides like merge-conflicts, permissions, authentication and some useful patterns, but the above code is about 90% of what you would expect a user to use in deepstream

https://vimeo.com/143728632 was what led to deepstream being initially written a few years back