Show HN: Pipedream. Develop any workflow, on any trigger, with auth and no infra
pipedream.com
pipedream.com
Pipedream is an integration platform built for developers. Develop any workflow, based on any trigger with authentication management built-in and no server or cloud resources to manage.
Workflows are code, which you can run for free.
The beta version includes the ability to:
- Run any Node code, or use pre-built actions
- Trigger workflows via HTTP, Cron, integrated apps or email
- Create, share, and fork workflows from the community and
- Send data to S3, Snowflake, email, SSE, and more.
Coming soon — Develop locally and deploy workflows via CLI, SDK, and more.
We think there’s a lot we can improve and are eager for feedback so please send us your ideas and opinions. Also, if you want a specific app or API fully integrated, let us know.
Here’s a screenshot to show what I mean: https://imgur.com/a/d0SjEVt
Reader view exposes the content fine, so that’s a valid workaround for now.
The constraints are listed in our docs - https://docs.pipedream.com/pricing/
No thanks, I've been there before. You'd be better off just charging from the get-go. Otherwise, this should be 100% open source.
I ask because I see a lot of these dismissive comments on HN which end with "open source it" - and that doesn't seem like a super constructive piece of advice to a new startup team.
FWIW, at my current company, an analytics company, about five years ago, we made it clear that no matter what data we collected, if you leave the service or we shut down, we will get you all that data to take with you in a very reasonable format.
It's always risky to invest in a new service, but sometimes risks bring great rewards, and I think it's helpful if you're giving input, to try to make it constructive.
If the model turns out to be inaccurate: A) Change the architecture to reduce costs to the point where it becomes profitable at an attractive price point for customers, or B) Move on to the next idea.
It's not that complex, and more startups should be more realistic about profitability.
That said, I get your point that if you can create a model that gives you a good sense of future profitability, you are in a better position not to die as a company.
This provides a signaling of value, creates a real business relationship, and directly contributes to the lifespan of the vendor.
A free model doesn't really tell their business if there's actually demand and if people actually value it when supply is "infinite" when in reality it isn't.
Guess a high price with some research for similar things and lower it iteratively. Def. see what sticks.
Also they could combine this with discounts, such as if someone preallocates capacity for X month, they get f(X) % discount.
They should build a core customer base. Solicit feedback about what niche feature they would happily pay for.
For example here's a GitLab issue that I think people would (might?) use: https://gitlab.com/gitlab-org/gitlab/issues/15536 (provide a "lock service" or a queue with max concurrency gate and launch pipelines / jobs)
The risk here seems small.
The value seems to be free hosting for the polling and scheduling.
This isn't too different than IFTTT abstraction-wise, and is easily reproducible in AWS using data pipelines and lambdas.
Why take an external dependency if what they are doing is rather simple?
I'd argue you are right about the complexity. The hook of the service seems to be easy integration and lack of infrastructure requirements.
At scale, a freemium model won't work. But it would provide initial traction for small use cases. This can be funded for a while without major rounds.
The advice to monetize now versus building community and getting users / crowd sourcing plugins is solid if the aim isn't to open source it.
With all workflow type services I’ve encountered I find that the actual underlying workflow can’t be managed under source control as one would do with (hopefully) most if not all application & infrastructure code.
Can I “deploy” workflows from source control?
Deploying workflows from source via CLI or your CI/CD pipeline is in the works! We want you to host workflows on Github / Gitlab and be able to deploy them using your standard process.
Feel free to reach out to me directly if you'd like more information and I can let you know when we're testing this.
dylan [at] pipedream [dot] com
BTW, love the name Pipedream!
I completely empathize with the time involved learning new tools. I'm happy to create a tutorial specific to your use case. We've built a few workflows to process Github events and I'd love to show you how this works end-to-end.
Is there anything specific you'd like to do with the commit / merge events, just so I make sure the tutorial targets your use case?
Feel free to reach out directly if you'd like to talk more at dylan [at] pipedream [dot] com.
The specific use-case I have in mind is that when a commit made I'd want to run various checks on configuration files in the repo to make sure specific conventions are followed and that certain file references exist. Upon a merge I'd like run a build process and send the build files to Heroku, Digital Ocean, or AWS.
https://www.youtube.com/watch?v=9ZGNP1-1vyg&feature=youtu.be
You can load the workflow I created for the video here:
https://pipedream.com/@dylburger/template-workflow-to-proces...
and press the Fork button in the top right to create a copy of it in your own account. You can modify that copy and run it for free.
The video isn't perfectly polished and I made a few mistakes, but we've built Pipedream to make it easy to debug your workflows, too, so I hope the mistakes help you understand part of its power!
A couple of notes on your specific use case:
* There's no webhook event for new commits. A PR (like I show in the video) or a push might be the best event to listen for. You can then run Node code to list commits associated with that push. See the API docs for commits here: https://developer.github.com/v3/repos/commits/ . * I wasn't sure specifically what build process you wanted to run, so I end the video reviewing how to add new steps to the workflow and walk through a few options: you can run any Node.js code or any of our pre-built actions. You have access to the /tmp dir on Pipedream and can use a package like child_process [1] to spawn some commands if you need to run anything on the shell. Unfortunately you don't have access to a full shell but let me know if what we provide works or doesn't — we'd love the feedback.
Let me know if that helps or if you have any questions!
[1] https://www.freecodecamp.org/news/node-js-child-processes-ev...
Great work! This is incredibly useful stuff and can't wait to start trying it out
Out of curiosity, are you hiring? I’d be interested in hearing what your longer-term plans are and am looking for something new. This seems right up my alley.
ETA: I see from the article that Python is indeed coming. I’d love to be a part of that!
We're planning to share more about our stack in future engineering blogs, so keep an eye out for those!
(Yes, Python is coming.)
+1
I'm a Pythonista at heart and we're working on Python support as we speak! We'll let y'all know when we ship that.
It's answered here[1], along with limitations[2]:
- 100kb max body-size.
- Rate-limit at 10 requests per second
- 10 seconds per execution for HTTP triggers, and 30 seconds for cron triggers
- 192 MB of memory for your code and libraries during workflow execution
- 512 MB of disk in the /tmp directory.
[1]https://docs.pipedream.com/pricing/ [2]https://docs.pipedream.com/limits/#limits
I have no desire to be hand-savvy so please ask whatever questions you have and I will do my best to clarify.
All of us were early employees of BrightRoll [1], an advertising marketplace that handled 10 million requests / second at peak. We've scaled large systems in the past and we've built this system with scale in mind.
Believe it or not, most workflows are extremely low volume. That, in part with the design of the system, allows us to offer the generous free tier.
We're planning to introduce paid tiers soon. You can see my comment on the parent thread that addresses some of the general questions around pricing asked by others in the thread.
This was a very early release of the platform and we hoped offering the product for free would encourage experimentation. To a great degree, we've seen that, so we think it was the right choice.
Testing and trying something new requires time and effort, which is limited, especially for companies that would be interested, if pricing/future isn't crystal clear, why make the effort?
ESPECIALLY since this would become a key part of the infrastructure / how my app gets its stuff done. Replacing your service with a competitor or something homebrew would be a major pain, especially when I find myself unexpectedly popular.
there is NO WAY I would ever use or recommend a service that refuses to tell me what my potential long term costs are up front.
It's too much of a gamble with a business or worse, a hobby project that isn't making me any money to compensate for this unexpected "balloon payment".
(I am not affiliated with this vendor, I just deal with software vendors a fair bit.)
1. https://www.flogo.io/ (Go)
2. https://n8n.io/ (Recently featured in HN https://news.ycombinator.com/item?id=21192132 )
I especially appreciate the discussion around how and why Pipedream is free, and the concerns around the lack of a visible, scalable paid tier. Those concerns are valid and I want to give you a little more insight into our thinking on this and what the future holds.
Workflows created by one developer can be forked, run, and modified by others. We all build a lot of the same integrations across companies and believe if this code is shared and executed on a common platform that’s purpose-built for running these workflows, it’ll save us all a lot of time.
We’ve worked with thousands of alpha users to understand how they’re using the product and considered what features enterprises ultimately will want and need to pay for. These include: higher workflow limits, private workflows and actions, SLAs, premium support, and more. We didn’t launch with that today because we’re focused on getting feedback from individual developers who will be the majority of users moving forward.
Of course, you can’t run a business solely on free workflows. We’ve set some limits on these workflows that help us control excessive use [1]. Our team has a wealth of collective experience scaling software companies on the business and tech side, and we have confidence that we’ll be able to retain a generous free tier while building a sustainable business.
I empathize with the skepticism of Pipedream and of hosted platforms in general, and welcome any more specific questions. We truly love feedback. We’ve implemented some ideas that we believe facilitate the developer experience for building workflows but are looking for y’all to test that, validate or invalidate it, and give us specific thoughts as you have time.
dylan [at] pipedream [dot] com
As a potential user, before even trying the tool I'd like to know how are you planning to monetize it. Free sounds awesome, but you'll get to a point where it costs you actual money to keep all this working. I don't want to waste my time creating pipelines in a tool that won't exist in a year or two -or that will become paid and break my existing workflows (or make me expend money unexpectedly)-.
[1]: https://n8n.io/
In the future, we may offer features available on paid tiers which would be logical enterprise features such as single sign on (SSO), team collaboration and higher SLAs and throughput.
The constraints are listed in our docs - https://docs.pipedream.com/pricing/
Learning a new tool -even if it's friendly and has a community around it- takes time and effort. We want to know it will be there in the future. That's why I'm interested to know: how are you planning to keep Pipedream alive for, let's say, the next 5 years?
Unfortunately, n8n is not[1] Open Source, despite claiming to be "Open Source and so free" on their website[2].
What about the license makes this not open-source?
>For purposes of the foregoing, “Sell” means practicing any or all of the rights granted to you under the License to provide to third parties, for a fee or other consideration (including without limitation fees for hosting or consulting/ support services related to the Software), a product or service whose value derives, entirely or substantially, from the functionality of the Software.
IANAL but sounds like that means if you're building apps for people on the platform, consulting, using it for clients/freelancing, or a hosted platform built on the technology then you're not licensed to do that and will need to get approval.
Q: Is this “Open Source”?
A: No.
I wish this article was a more digestible single case as an introduction that showed value rather than saying 'you can run anything!!! for free!!!'
The post is meant to articulate the depth of our vision for the platform. The hope was the linked workflows provide some inspiration and examples to make Pipedream a little more concrete. I'm glad the homepage resonated with you.
Did you get a chance to sign up and use the tool? We're looking for specific feedback from devs on how we can improve so I'd love if you had any time to build a workflow and share your thoughts.
How do you differentiate yourself. Do you (or will you) support frameworks like Serverless?
We've found that for integrations or common automations, building on a cloud platform is too low-level. You've got to manage a number of services that aren't core to your use case (e.g. API Gateway, IAM, Cloudwatch Logs, etc.). We operate at a higher level and just give you an execution environment to run Node.js code or stitch together pre-built actions, similar to integration platforms like Zapier.
Of course, Serverless and other frameworks abstract part of the cloud services, but you're still running on that platform, and when you need to troubleshoot or scale the service, it's likely you'll have to peek under the covers. We tried to build a serverless platform from the ground up and provide services like the HTTP server, key-value store, logs, and other services right inside the UI, or accessible in code. There's no other cloud resources to manage or monitor.
This kind of abstraction isn't great for every application, but we think it shines when you're building integrations or data pipelines that aren't part of your core app.
The pre-built actions and built-in OAuth / key-based auth also operate at a higher level than most cloud services. We've tried to provide these in a developer-friendly way, so you have full control over how your workflow runs, but don't have to manage the parts that are tangential to the logic of your app.
As we expose more APIs for building and running workflows, we'd love to support a framework like Serverless.
I'd love if you have time to give Pipedream a try and see how it works for yourself! We're looking for feedback on how we can improve so we'd love more eyes on it.
If you've used it and like it, I'd honestly love for you to give Pipedream a try and tell us where we can improve compared to your experience with other tools. We'd love the feedback!
dylan [at] pipedream [dot] com
I'm a founder of a startup in the same space, so just wanted to say good luck. I love the name. It's exciting to see so many folks working on making connective developer tooling a better experience.
I had a link originally included in here but -- you guys should enjoy your launch. I understand and have a great deal of empathy for the effort it takes to build and launch something new. Enjoy all of it, and keep up the good work.
- Keith
Your code runs in its own Node execution environment and VM, completely isolated from other users' code.
Happy to answer any other specific questions you have! Feel free to reach me at dylan [at] pipedream [dot] com .
As far as I can see the above should come with a footnote saying "for some value of any".
Looks very cool, despite being entirely web-centric.
dylan [at] pipedream [dot] com
Only read the landing page so far but it looks like Lambda with just Node support. Can someone correct me if I’m off?
We've found that for integrations or common automations, building on a cloud platform is too low-level. Frequently you're working with more than just a Lambda. You've got to manage a number of services that aren't core to your use case (e.g. API Gateway, IAM, Cloudwatch Logs, etc.). There's a lot of boilerplate config, and many of these services are tangential to the core logic of your app.
We operate at a higher level and just give you an execution environment to run Node.js code or stitch together pre-built actions, similar to integration platforms like Zapier. We manage the HTTP endpoint, function config, and give you built-in observability. We also provide higher-level services, like built-in app integrations and OAuth / API key-based authorization. Building this yourself on AWS can take time.
This kind of abstraction isn't great for every application, but we think it shines when you're building integrations between services.
I'd love if you had time to sign up and see how this works! It's free to use and we'd love the feedback.
Let me know if that answers your questions, or if you have any more!
I mean, you can write code in-line with AWS Lambda but to have an application help setup multiple Lambdas and/or integrate outside APIs would be pretty neat.
I'd love if you had the chance to sign up. It's free to use and we'd love the feedback.
We've found that for integrations or common automations, building on a cloud platform is too low-level. Frequently you're working with more than just a Lambda. You've got to manage a number of services that aren't core to your use case (e.g. API Gateway, IAM, Cloudwatch Logs, etc.). There's a lot of boilerplate config, and many of these services are tangential to the core logic of your app.
We operate at a higher level and just give you an execution environment to run Node.js code or stitch together pre-built actions, similar to integration platforms like Zapier. We manage the HTTP endpoint, function config, and give you built-in observability. We also provide higher-level services, like built-in app integrations and OAuth / API key-based authorization. Building this yourself on AWS can take time.
This kind of abstraction isn't great for every application, but we think it shines when you're building integrations between services.
Let me know if that answers your questions, or if you have any more!
The constraints are listed in our docs - https://docs.pipedream.com/pricing/
PS: On Desktop, the left third of the Pricing content/copy is rendered behind the menu.
We use Vuepress for the docs and it's awesome!