Open
37Under consideration
[Template] A brief title that clearly describe what the feature is about
Content: A detailed explanation on how you want the feature to work and what is the use case of the feature? If possible, include any image references, videos, screenshots, or sketches to help convey your ideas more effectively. Impact of Not Having This Feature: What are the consequences of not implementing this feature. Will it hinder your or other users' ability to use the product effectively? [Example] Title: Enable weave to be used as a plugin on social media manager tools Content: As a social media manager, I'm using Later to plan and manage the social media post. Currently, I'm using Later to pre-schedule the post and use some other content generator tools to generate the contents and have to manually upload the content to Later. I find that Weave is super helpful for content generation and wish that there could be a Weave plugin on Later to create seamless content management.
Automated LLM Listing System
The feature is about building a LLM Listing system, whereby the listings of LLMs is managed and automated based on approval from the back office. This would allow Weave users to add any LLMs on their own. Users can add custom LLM by just inserting API url, key and documentations url. Weave BE will scrape the docs, build the manifest and store in on db. Impact of not having this feature This feature doesn't affect it's functionality but will add value to Weave as more LLMs can be supported
Add prompt injection prevention mechanism
We should add in prompt injection prevention mechanism: • OpenAI moderation (free) undefined: https://platform.openai.com/docs/guides/moderation/overview • Prompt injection prevention from Bittensor SN14 undefined: https://docs.synapsec.ai Johnson got the api key from http://synapsec.ai as well
Making Weave API OpenAI compatible
With weave api being open ai compatible, user can just change their API in just 2 lines, then we can onboard user that used OpenAI quickly. const openai = new OpenAI({ baseURL: "https://pms.chasm.net/api/prompts/execute/44", apiKey: OPENAI_API_KEY, });``` Also with this feature, we can integrate with Langchain easily as well, user can just change model through API Slack Discussion: https://chasm-talk.slack.com/archives/C05PHU191GX/p1715774841176749
Revert feature
Currently, there is the draft feature which allows users to edit prompts while the API will always be running on the saved version. If the user decides that they want to discard their draft version and revert to the saved version, there is no button/feature for that.
Enable users to share workflow as a template
There could be two sharing scenarios: Pure sharing of the workflow, where the receivers can make a copy from the workflow, and edit it as his own version Create a collaborative team to manage the workflow together No 2. would be very useful, if in the future, we've more business users.
Method to process the output for a prompt
Why do we need this? Perform the formatting of the output before it get stored into a variable to be used by following prompts String formulas - appending text? Extracting values of the prompt output before it get stored into a variable Perform operations on numerical outputs
[Weave] Extractable JSON from Prompt responses
When a prompt produces an output, there can be a step or automated action to extract the keys into a workflow variable
[Weave] Agents
Agents are super powerful to be used in LLM in order for LLM to use a certain tools. Some reverse engineering of Agents: https://www.notion.so/hoogagaming/Agents-e26775253c5145159fe587e162432f9d?pvs=4 What you can achieve w Agent is that let say you ask "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?" using LLM without Agent, it will gives u data until 2021 and maybe not be capable of doing Math, which might be wrong. With Agents (LLM Math & Serp API), the prompt will be breakdown into tasks, and it will finds out through internet who's Leo's gf then get her age. After that it will use LLM Math to calculate the value. The UI can be just simple box for users to tick Then we can just focusing on building Agents plugin
[Weave] Image prompt flow.
btw we’ve been meeting a lot of ppl in sg, I think our image prompt could be one feature we can add in, can secure a few users already hahaaaa ^ Note, lower priority compared to most PMS centric features
Suggest to add a Update Password UI for users who sign up with Google
For users who sign up with Google, there is no way to update their password except for using the forgot password. It might be good to have add a UI for user to update password.
In Review
16Under consideration
Conditional or Logic block to enable Workflow brances
In order to make a workflow more comprehensive to meet different business needs, there will be a need to allow different paths based on the workflow variable or value
Notification System for Users
Notification channels 1. Email 2. Platform notifications 3. In page notifications Features: 1. Allow user to opt in/out from email notifications 2. Allow user to read the notifications in Weave What notifications? Model is retired New model is added Credit balance running low etc
Starter pack templates for users' reference
We should include a list of templates that meets the general use cases of the users to start their prompting or workflow usage.
Placeholder icon and page for fine tuning
maybe we can add a pending page thingy for automated fine tuning soon? As a placeholder icon right after prompt playground and workflow builder
Teams Function for collaborative feature
Content: This feature is aimed at making teamwork easier on the platform by letting multiple team members work together on the same prompt and same workflow. This helps in organising teamwork and centralised communication. Under a new menu titled "Teams", I propose the following options: Create New Team This allows a user to start a new team, name the team, and add some members to start with. Invite New Users into Team This option lets team leaders or allowed members invite more people to join the team using their email. Remove Member from Team This will let authorized persons remove a member from the team and, making sure the team's work stays safe.
[Weave] Domain Whitelisting
For security purposes, to avoid hackers to tamper the API keys and access to the endpoints
[Weave] Internet Connection for LLM
Currently most LLM doesnt have internet connection, the trained data only until 2021, with this it is really useful for a better results with real time data. we can use API product like SerpAPI.
[Weave] Default prompt template insertion in UX flow.
Unsure if this is already a pending feature, but having a default prompt template inserted when the user selects one of the llama2 models would be great. From a UX flow, the default prompt template should appear when one of the Llama models is selected, and disappear when an alternative model is selected.
[Weave] Feature: Prompt playground response time from LLM
Feature: Prompt playground response time from LLM - would be great to know how much time a prompt spent to run with that specific LLM. Helps users to properly decide between models as well as structure prompt complexity https://chasm-talk.slack.com/archives/C05PHU191GX/p1695637378933809 Discussion log ^
Workflow versioning
Allow users to Save a workflow as a draft Publish a version of the workflow Copy a version as draft Roll back to previous version of workflow This is for users to backtrack their changes
Planned
6Committed and queued
Reorganizing AI Models in 'All' view
Description: Currently, the arrangement of AI models in the All models menu is scattered. Models from the same provider not grouped together. For example, Meta has models on the top on the list and also at the bottom of the list. This setup can lead to confusion and inefficiency when users are trying to locate specific models. Proposed Change: Reorganising the AI models by sequencing them according to their respective providers. This would entail placing all LLM models in a contiguous block. This will improve navigational efficiency and user experience by making it easier to find and compare models from the same provider.
Duplicate Prompts & Workflows
Context: Currently there is no option of deleting or duplicating created prompts and workflows. Why is it beneficial? It would be useful for people who wanted to test slightly different versions of prompts / workflow by duplicating them quickly, or delete unwanted prompts / workflows.
Improve API panel to help developer better understand on the usage
Devs are having hard time to understand how to properly use the API. Suggestions: Increase the visual hierarchy & focus of endpoint URL, payload structure, response data structure. Reduce the language options (currently we have multiple implementation for each language, but actually that doesn't help much)
As a user, I want to be able to sort the prompts list in the workflow module
As a user, I want to be able to sort the prompts in the prompts list according to: last modified date created date last published date
Show status for prompt node in workflow
When a prompt has unpublished changes, show an indicator on the prompt node to inform users in the Workflow module
Sticky note in the Workflow Module
Suggestion: To have a sticky note for jotting down ideas, reminders or for better collaborations. How it looks like: https://www.figma.com/file/6ENzA3PXIZEgUlJ0MxjGKm/Weave-(MVP-%E2%80%93-Q4-2023)?type=design&node-id=640%3A2432&mode=design&t=WRmgxuBncJd9fdIR-1
PRD in Progress
1Actively being built
Dev In Progress
6Actively being built
Reset the parameter settings in one click
Issue: After changing the values for the parameters, it is difficult to reset the value back to default as there is no obvious indicator. Current implementation is to double click on the middle of the bar but it is too difficult to manoeuvre. There is also no tooltips to teach user to do this. Suggested implementation: Have a one-click reset all button at the top to reset all values to default. At each line, have a small button at each of the parameter to reset only that particular value to default.
Delete Prompt or Workflow
Allow user to delete prompt or workflow in Weave
Filter function for core modules
As a user, I want to be able to filter prompts / workflow by status As a uesr, I want to be able to filter datasets by file type
Sorting function for core modules
As a user, I want to be able to sort my prompts, workflow, datasets by: name (alphabetical order or reverse) created date (earliest to latest, or reverse) updated date (latest to earliest, or reverse)
Option for users to view core modules in grid or list view
By default, users can view the core modules in grid view. Users can have the option to toggle between grid view or list view. Core modules include: Prompt, Workflow and Datast
Search function for all core modules
As a user, I want to be able to search for a prompt by title As a user, I want to be able to search for a workflow by title As a user, I want to be able to search for a dataset by title
Completed
29Recently shipped
[Weave] Using the output of a prompt as a input for other prompts in the workflow
Currently there is no way to pull the output from the first prompt in the workflow, and insert it into the next prompt. “How do I prompt in a way that the output of the first process will be used as the input of the second process?” This is a basic and super important feature that needs to be inserted into the workflow, so that we can start testing.
Dataset & RAG Module
Dataset module will consists of datasets/files that can be used within prompts. Dataset can be in the form of txt, pdf and etc. https://hooga-gaming.atlassian.net/wiki/spaces/Product/pages/155811853/Weave+-+PRD+-+RAG+Doc+Processing+and+Dataset+Module
GPT-4o
I think we should add gpt-4o since the price of it is cheaper than gpt-4 and gpt-4-turbo Link: https://openai.com/api/pricing/
Allow Output to support Emoji
When the actual output has emoji, Weave output doesn't support the emoji display. Refer to attachment Package link: https://uiwjs.github.io/react-json-view/ Slack Thread: https://chasm-talk.slack.com/archives/C05PHU191GX/p1701579288519599
Periodic Health Check on LLM's
As per what Azad mentioned too above, having a health checkon the api (undefined: http://34.102.119.218:6507/health) , (undefined: http://34.102.119.218:6570/health) etc etc would be great.
Enhanced Messaging Types Support
Background: Our current product lacks the support for System Messages in ChatGPT/Llama2 and requires a custom instruction mechanism [INST]<<SYS>> [/INST] for Llama2. The absence of a structured messaging type framework restricts the flexibility and interaction model that can be crucial for developer-centric applications. Site like ChatGPT Playground https://platform.openai.com/playground Benefits: Improved interaction: This message types allow more interactive and intuitive communication model Better UX: For Llama2 user, user dont need to type in [INST]<<SYS>> any more, also allow user to swap LLM easily Future-proofing: ChatGPT playground already shown usecases Proposal: Support Multiple message and Message Type like System, User, Assistant for both ChatGPT and Llama2 Support extendable API for message type to allow conversational usecase, useful for things like ChatBot / Interview AI { "input": { "question":"USA" }, "messages": [{ "assistant": "USA is a ..." }, { "user": "How about Mexico?" }] } Technical ChatGPT From ChatGPT-3.5-turbo onwards they are supporting multiple messages {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"}, {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."}, {"role": "user", "content": "Where was it played?"} https://platform.openai.com/docs/guides/gpt/chat-completions-api Llama2 Consideration / Open Questions How to handle maximum token size? Advance user want to customize Llama2, how can they do it?
[Weave] Accounting System
To account for API Keys & Credits consumption
Qwen1.5
Name of AI model: Qwen1.5 Huggingface / API Documentation Links: https://huggingface.co/collections/Qwen/qwen15-65c0a2f577b1ecb76d786524 https://openrouter.ai/models?q=qwen https://www.reddit.com/r/LocalLLaMA/comments/19e9xfj/introducing_qwen/ Purpose of using this model: Good for Mandarin and Japanese language inferencing, apparently has beaten GPT4 turbo for understanding Mandarin. Could be useful for those foreign language LLM requirements as a viable alternative to GPT. Issues involving hyperalignment as it's trained by distilling GPT lol. Current use cases that is already using this model Coding (surprisingly), different chatbots for different languages etc etc. Multimodal it's possible to have other input/output forms like visual and audio too.
Model Catalog
Before user selects a model to run their prompt, the model catalog can help the users by reading the description of the model to help them better understand of the available model shortlist the LLM models according to their use case and preference searching for the model for the ease of finding their preferred model This model catalog is also considered as part of the expansion plan where Weave will be the aggregator of dAi models
As a user, I want to be able to search for prompts in the prompts side panel in Workflow module
Use case As a user, I want to be able to search for prompts in the prompts side panel under the workflow module Current Behaviour In workflow module, when user types in the search bar in the left side panel, it doesn’t bring up any prompts. Expected Behaviour When user types the name of the prompt in the search bar, it should bring up related prompts.