243 karma · joined January 26, 2020
Testing considerations for LLM tasks https://www.promptotype.io/blog/posts/testing-considerations
An overlooked game-changing use case of LLMs for software developers https://www.promptotype.io/blog/posts/llm-structured-tasks
Here's the story:
At first I was building a tool for stock analysis- the user writes in free language what companies they want to compare, along with a time period, and their requested stocks show up on a graph. They can then further reiterate on it- add companies, and change range all in free language (I had many more analysis functions planned). Following some unique dev challenges I've found- I ended up not releasing the product (possibly will sometime in the future..), and switched to work on a dev platform to help with these challenges.
I was using what I called 'LLM structured task'- basically instructing the LLM to perform some task on the user input, and outputting a json that my backend can work with (in the described case- finding mentioned companies and optional time range, and returning stock symbols, and string formatted dates). The prompting has turned out to be not trivial, and kind of fragile- things broke with even minor iterations on the prompt or model configurations. So- I developed a platform to help with that- testing (templated) prompt versions, as well on model configurations on whole collections of inputs at once- making sure nothing breaks in the development process (or after). * If you're interested, welcome to check it out on https://www.promptotype.io
Other than that, it's all a function of compensation/risk, safety, and effort compared to their other options.
Completely customizable tracking: t12n is intended to be very open, in the way that you can track and follow up on literally whatever you want (you're the one defining the tracking questions, and the responses to them).
Experimentation: t12n allows you to define timed experiments of behavior changes, and measure their effects on your tracking- allowing you to consequently decide which behavior changes are actually worth keeping for the long run.
Analytics: T12n provides you with analytics data that's intended to provide you with insights on your tracking.
Last (and very importantly)- notification based tracking: imho an actual game changer, in the way that it minimizes the tracking friction as much as possible: you can be notified whenever you want, and respond to the tracking question directly from the notification- literally takes less than 5 seconds!
Lets you track whatever you want by responding directly to the notification, on your phone or watch.
Also provides general analytics and measures experiments effects (how do time periods with new behaviors affect your tracking).
Have been using it myself for a while now, and will soon post further about my own learnings from it.
May I ask how do you use some of these outputs? Is it mostly for the action of journaling itself? do you feel like it helps you optimize yourself/ your day to day (referring to the stuff you didn't mention the functionality of such as lazy tracking of health data, personal notes that don't relate to human relationships etc)?
> Or, ask a professional, i.e., a doctor. I'll refer to his first, of course I'll go to the doctor for anything serious :) I'm trying to optimize for less severe things that can be VERY difficult to diagnose, if only because they can have so many reasons causing them, and the lack of resources due to their low severity.
> canvas the literature for some causal hypotheses based in what we already know, and then design the experiment This is something I'm trying to do, but was looking into a better way to track and follow up on. I was thinking for example, it can be potentially easy to see a correlation between back pains and a new type of work out I started to do. I'm trying to track these kind of things and see if I find correlations (will try better, automated analysis soon too).
Thanks for the reference for Quest Diagnostics, will check it out!
Cool system you have implemented there! May I ask if you have a way to process and analyze the data? Do you feel a need to?
wrt to voice assistant automation- someone mentioned here in the comments they use the Google assistant to add stuff to keep or notes, maybe you can try using something like this, combined with another automation to store the data in your preferred structure? (I tried something similar before developing the app, but without using voice assistant and with possibly simpler, structured data and it worked well).
Are you thinking of a way to automate the actual tracking? Do you find it a hassle to do manually? (this is the main thing I was trying to solve, check out https://www.t12n.io )
Can you share what do you mean exactly by "more structured"? I would expect Trello to be more structured then Joplin, what am I missing?
Not sure I understood what you connect to Evernote with Zapier to track vitamin D usage and mood (or was Zapier only relevant for the weather and the others are manual?). Also, if it's ok to ask- what's the ephemeral value in tracking these?
Were the sheets helpful in pinpointing the cause for an illness (or you stopped because they weren't)? If so, interesting to hear what method you used to do that (from the raw data).